[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from ClientFailoverOnNegotiationTimeoutITest
[ RUN      ] ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer
2026-08-12T08:02:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:16Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 08:02:16.170534 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:42055
--webserver_interface=127.22.155.62
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:42055
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
W20260812 08:02:16.275148 23163 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:16.275465 23163 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:16.275512 23163 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:16.278906 23163 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:16.278985 23163 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:16.279004 23163 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:16.279024 23163 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:16.279042 23163 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:16.282872 23163 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:42055
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.155.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:16.283989 23163 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:16.285076 23163 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:16.290984 23170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.290966 23169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.290925 23172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:16.291039 23163 server_base.cc:1061] running on GCE node
I20260812 08:02:16.291662 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:16.292598 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:16.293779 23163 hybrid_clock.cc:648] HybridClock initialized: now 1786521736293732 us; error 60 us; skew 500 ppm
I20260812 08:02:16.295672 23163 webserver.cc:533] Webserver started at http://127.22.155.62:36955/ using document root <none> and password file <none>
I20260812 08:02:16.296231 23163 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:16.296289 23163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:16.296623 23163 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:16.298357 23163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data/instance:
uuid: "9c6920d259724cf6aa378e67fed1bb30"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.298926 23163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal/instance:
uuid: "9c6920d259724cf6aa378e67fed1bb30"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.302379 23163 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:02:16.304478 23183 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.305455 23163 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:16.305599 23163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal
uuid: "9c6920d259724cf6aa378e67fed1bb30"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.305703 23163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:16.327435 23163 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:16.328109 23163 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:16.328310 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:16.335884 23163 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.62:42055
I20260812 08:02:16.335920 23270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.62:42055 every 8 connection(s)
I20260812 08:02:16.336896 23163 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:16.339723 23272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:16.344470 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30: Bootstrap starting.
I20260812 08:02:16.346835 23272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:16.347121 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 23163
I20260812 08:02:16.347257 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/master-0/wal/instance
I20260812 08:02:16.347738 23272 log.cc:826] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:16.349747 23272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30: No bootstrap required, opened a new log
I20260812 08:02:16.352684 23272 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } }
I20260812 08:02:16.352872 23272 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:16.352943 23272 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6920d259724cf6aa378e67fed1bb30, State: Initialized, Role: FOLLOWER
I20260812 08:02:16.353360 23272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } }
I20260812 08:02:16.353468 23272 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:02:16.353559 23272 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:02:16.353681 23272 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:16.354509 23272 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } }
I20260812 08:02:16.354853 23272 leader_election.cc:304] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c6920d259724cf6aa378e67fed1bb30; no voters: 
I20260812 08:02:16.355145 23272 leader_election.cc:290] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:02:16.355253 23277 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:16.355526 23277 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6920d259724cf6aa378e67fed1bb30, State: Running, Role: LEADER
I20260812 08:02:16.355880 23277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } }
I20260812 08:02:16.356251 23272 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:16.357012 23278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c6920d259724cf6aa378e67fed1bb30. Latest consensus state: current_term: 1 leader_uuid: "9c6920d259724cf6aa378e67fed1bb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } } }
I20260812 08:02:16.357125 23278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:16.357509 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c6920d259724cf6aa378e67fed1bb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6920d259724cf6aa378e67fed1bb30" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 42055 } } }
I20260812 08:02:16.357615 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:16.359455 23292 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:16.362044 23292 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:16.367298 23292 catalog_manager.cc:1383] Generated new cluster ID: cf08f94fea444fffb5e6cf10b40bf428
I20260812 08:02:16.367420 23292 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:16.378073 23292 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:16.378810 23292 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:16.387905 23292 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 9c6920d259724cf6aa378e67fed1bb30: Generated new TSK 0
I20260812 08:02:16.388422 23292 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:16.394767 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.1:0
--local_ip_for_outbound_sockets=127.22.155.1
--webserver_interface=127.22.155.1
--webserver_port=0
--tserver_master_addrs=127.22.155.62:42055
--builtin_ntp_servers=127.22.155.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
W20260812 08:02:16.509491 23312 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:16.509814 23312 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:16.509944 23312 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:16.513441 23312 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:16.513543 23312 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:16.513638 23312 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:16.513734 23312 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.1
I20260812 08:02:16.517575 23312 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.155.1
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:42055
--never_fsync=true
--heap_profile_path=/tmp/kudu.23312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:16.518640 23312 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:16.519732 23312 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:16.522441 23312 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:16.526080 23320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.526162 23325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.526227 23321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:16.526511 23312 server_base.cc:1061] running on GCE node
I20260812 08:02:16.526912 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:16.527482 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:16.528683 23312 hybrid_clock.cc:648] HybridClock initialized: now 1786521736528671 us; error 28 us; skew 500 ppm
I20260812 08:02:16.530829 23312 webserver.cc:533] Webserver started at http://127.22.155.1:39795/ using document root <none> and password file <none>
I20260812 08:02:16.531387 23312 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:16.531471 23312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:16.531679 23312 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:16.533409 23312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data/instance:
uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.533988 23312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance:
uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.537510 23312 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:02:16.539664 23333 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.540680 23312 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:16.540800 23312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal
uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.540935 23312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:16.568840 23312 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:16.569625 23312 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:16.569830 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:16.571673 23312 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:16.571727 23312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.571765 23312 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:16.571805 23312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.583665 23312 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.1:46399
I20260812 08:02:16.583688 23537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.1:46399 every 8 connection(s)
I20260812 08:02:16.584594 23312 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:16.586016 23539 heartbeater.cc:344] Connected to a master server at 127.22.155.62:42055
I20260812 08:02:16.586196 23539 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:16.586714 23539 heartbeater.cc:507] Master 127.22.155.62:42055 requested a full tablet report, sending...
I20260812 08:02:16.588075 23210 ts_manager.cc:194] Registered new tserver with Master: 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:16.589504 23210 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.1:44243
I20260812 08:02:16.591889 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 23312
I20260812 08:02:16.592015 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance
I20260812 08:02:16.597398 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.2:0
--local_ip_for_outbound_sockets=127.22.155.2
--webserver_interface=127.22.155.2
--webserver_port=0
--tserver_master_addrs=127.22.155.62:42055
--builtin_ntp_servers=127.22.155.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
I20260812 08:02:16.616677 23539 heartbeater.cc:499] Master 127.22.155.62:42055 was elected leader, sending a full tablet report...
W20260812 08:02:16.704229 23541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:16.704414 23541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:16.704461 23541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:16.707646 23541 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:16.707706 23541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:16.707746 23541 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:16.707787 23541 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.2
I20260812 08:02:16.711472 23541 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.155.2
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:42055
--never_fsync=true
--heap_profile_path=/tmp/kudu.23541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.2

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:16.712373 23541 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:16.713408 23541 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:16.715941 23541 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:16.719483 23551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.719493 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.719499 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:16.719905 23541 server_base.cc:1061] running on GCE node
I20260812 08:02:16.720247 23541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:16.720813 23541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:16.721993 23541 hybrid_clock.cc:648] HybridClock initialized: now 1786521736721989 us; error 42 us; skew 500 ppm
I20260812 08:02:16.723709 23541 webserver.cc:533] Webserver started at http://127.22.155.2:39259/ using document root <none> and password file <none>
I20260812 08:02:16.724335 23541 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:16.724421 23541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:16.724643 23541 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:16.726320 23541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data/instance:
uuid: "9c1d86a754e645269bc0796059331944"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.726886 23541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1d86a754e645269bc0796059331944"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.729990 23541 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:02:16.731930 23564 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.732868 23541 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 08:02:16.732999 23541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal
uuid: "9c1d86a754e645269bc0796059331944"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.733075 23541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:16.747447 23541 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:16.747938 23541 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:16.748073 23541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:16.749666 23541 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:16.749735 23541 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.749796 23541 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:16.749872 23541 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260812 08:02:16.761125 23541 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.2:33271
I20260812 08:02:16.761421 23774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.2:33271 every 8 connection(s)
I20260812 08:02:16.762153 23541 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
I20260812 08:02:16.763334 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 23541
I20260812 08:02:16.763408 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-1/wal/instance
I20260812 08:02:16.765002 23775 heartbeater.cc:344] Connected to a master server at 127.22.155.62:42055
I20260812 08:02:16.765153 23775 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:16.765563 23775 heartbeater.cc:507] Master 127.22.155.62:42055 requested a full tablet report, sending...
I20260812 08:02:16.766484 23211 ts_manager.cc:194] Registered new tserver with Master: 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:16.767021 23211 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.2:50513
I20260812 08:02:16.769299 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.3:0
--local_ip_for_outbound_sockets=127.22.155.3
--webserver_interface=127.22.155.3
--webserver_port=0
--tserver_master_addrs=127.22.155.62:42055
--builtin_ntp_servers=127.22.155.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
I20260812 08:02:16.793664 23775 heartbeater.cc:499] Master 127.22.155.62:42055 was elected leader, sending a full tablet report...
W20260812 08:02:16.874446 23778 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:16.874737 23778 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:16.874836 23778 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:16.878376 23778 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:16.878474 23778 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:16.878554 23778 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:16.878652 23778 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.3
I20260812 08:02:16.882504 23778 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.155.3
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:42055
--never_fsync=true
--heap_profile_path=/tmp/kudu.23778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.3

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:16.883633 23778 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:16.885037 23778 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:16.887787 23778 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:16.891991 23790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.892027 23788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:16.891999 23787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:16.892336 23778 server_base.cc:1061] running on GCE node
I20260812 08:02:16.893420 23778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:16.894117 23778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:16.895352 23778 hybrid_clock.cc:648] HybridClock initialized: now 1786521736895275 us; error 99 us; skew 500 ppm
I20260812 08:02:16.897372 23778 webserver.cc:533] Webserver started at http://127.22.155.3:36359/ using document root <none> and password file <none>
I20260812 08:02:16.897913 23778 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:16.897967 23778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:16.898118 23778 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:16.899600 23778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data/instance:
uuid: "8811a4670c574f729b04cebef39bc7aa"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.900167 23778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal/instance:
uuid: "8811a4670c574f729b04cebef39bc7aa"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.903769 23778 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 08:02:16.906126 23799 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.907181 23778 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:02:16.907341 23778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal
uuid: "8811a4670c574f729b04cebef39bc7aa"
format_stamp: "Formatted at 2026-08-12 08:02:16 on dist-test-slave-ffrd"
I20260812 08:02:16.907488 23778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:16.925729 23778 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:16.926424 23778 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:16.926617 23778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:16.928550 23778 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:16.928625 23778 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.928690 23778 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:16.928720 23778 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:16.941164 23778 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.3:44619
I20260812 08:02:16.941475 24004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.3:44619 every 8 connection(s)
I20260812 08:02:16.942200 23778 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
I20260812 08:02:16.943620 24005 heartbeater.cc:344] Connected to a master server at 127.22.155.62:42055
I20260812 08:02:16.943811 24005 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:16.944235 24005 heartbeater.cc:507] Master 127.22.155.62:42055 requested a full tablet report, sending...
I20260812 08:02:16.945258 23211 ts_manager.cc:194] Registered new tserver with Master: 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:16.945556 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 23778
I20260812 08:02:16.945667 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer.1786521736143602-23148-0/minicluster-data/ts-2/wal/instance
I20260812 08:02:16.945996 23211 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.3:51217
I20260812 08:02:16.954643 23148 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:16.969012 23211 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43016:
name: "kudu1580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "val"
    type: INT32
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20260812 08:02:16.971590 23211 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu1580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:02:16.972733 24005 heartbeater.cc:499] Master 127.22.155.62:42055 was elected leader, sending a full tablet report...
I20260812 08:02:16.992538 23416 tablet_service.cc:1511] Processing CreateTablet for tablet e82eb9371afc404ca426b35866b8fc52 (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:16.992274 23885 tablet_service.cc:1511] Processing CreateTablet for tablet 21ed101d57de4eb983765b92682e2b5f (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:16.992826 23883 tablet_service.cc:1511] Processing CreateTablet for tablet 88f4b4af90a54b149dd1c350d0040d7c (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:16.993080 23884 tablet_service.cc:1511] Processing CreateTablet for tablet e82eb9371afc404ca426b35866b8fc52 (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:16.993387 23416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82eb9371afc404ca426b35866b8fc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:16.993414 23885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ed101d57de4eb983765b92682e2b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:16.997956 23884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82eb9371afc404ca426b35866b8fc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:16.998150 23883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f4b4af90a54b149dd1c350d0040d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:16.999284 23643 tablet_service.cc:1511] Processing CreateTablet for tablet 21ed101d57de4eb983765b92682e2b5f (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:17.000258 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ed101d57de4eb983765b92682e2b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:17.001443 23646 tablet_service.cc:1511] Processing CreateTablet for tablet e82eb9371afc404ca426b35866b8fc52 (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:17.001739 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82eb9371afc404ca426b35866b8fc52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:17.002323 24031 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: Bootstrap starting.
I20260812 08:02:17.005062 24031 tablet_bootstrap.cc:654] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.005450 23645 tablet_service.cc:1511] Processing CreateTablet for tablet 88f4b4af90a54b149dd1c350d0040d7c (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:17.005743 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f4b4af90a54b149dd1c350d0040d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:17.005949 24031 log.cc:826] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:16.992281 23412 tablet_service.cc:1511] Processing CreateTablet for tablet 88f4b4af90a54b149dd1c350d0040d7c (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:17.007802 24031 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: No bootstrap required, opened a new log
I20260812 08:02:17.007822 23412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f4b4af90a54b149dd1c350d0040d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:17.008003 24031 ts_tablet_manager.cc:1403] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 08:02:16.992274 23417 tablet_service.cc:1511] Processing CreateTablet for tablet 21ed101d57de4eb983765b92682e2b5f (DEFAULT_TABLE table=kudu1580 [id=e3c5c952eeab4bf7a3f50157429b79a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:17.009404 23417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ed101d57de4eb983765b92682e2b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:17.010915 24039 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: Bootstrap starting.
I20260812 08:02:17.011147 24031 raft_consensus.cc:359] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.011371 24031 raft_consensus.cc:385] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.011449 24031 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.011896 24031 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.012831 24031 ts_tablet_manager.cc:1434] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:02:17.013928 24039 tablet_bootstrap.cc:654] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.014288 24031 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa: Bootstrap starting.
I20260812 08:02:17.015426 24031 tablet_bootstrap.cc:654] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.016517 24042 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: Bootstrap starting.
I20260812 08:02:17.016809 24039 log.cc:826] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:17.018589 24039 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: No bootstrap required, opened a new log
I20260812 08:02:17.018715 24042 tablet_bootstrap.cc:654] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.018776 24039 ts_tablet_manager.cc:1403] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20260812 08:02:17.019912 24042 log.cc:826] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:17.021327 24031 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa: No bootstrap required, opened a new log
I20260812 08:02:17.021437 24031 ts_tablet_manager.cc:1403] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.000s
I20260812 08:02:17.021540 24042 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: No bootstrap required, opened a new log
I20260812 08:02:17.021725 24042 ts_tablet_manager.cc:1403] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:02:17.021983 24031 raft_consensus.cc:359] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.022100 24031 raft_consensus.cc:385] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.022147 24031 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.022040 24039 raft_consensus.cc:359] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.022269 24039 raft_consensus.cc:385] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.022291 24031 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.022390 24039 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.022583 24031 ts_tablet_manager.cc:1434] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20260812 08:02:17.022753 24031 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa: Bootstrap starting.
I20260812 08:02:17.022868 24039 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.023751 24031 tablet_bootstrap.cc:654] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.023912 24039 ts_tablet_manager.cc:1434] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:02:17.025439 24041 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:17.025591 24041 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.026255 24042 raft_consensus.cc:359] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.026536 24042 raft_consensus.cc:385] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.026635 24042 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.026782 24041 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:17.027184 24042 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.027361 24031 tablet_bootstrap.cc:492] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa: No bootstrap required, opened a new log
I20260812 08:02:17.027460 24031 ts_tablet_manager.cc:1403] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.000s
I20260812 08:02:17.027945 24031 raft_consensus.cc:359] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.028056 24031 raft_consensus.cc:385] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.028096 24031 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.028252 24031 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.028343 24042 ts_tablet_manager.cc:1434] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260812 08:02:17.029009 24031 ts_tablet_manager.cc:1434] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:02:17.026741 24039 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9: Bootstrap starting.
I20260812 08:02:17.030246 24042 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: Bootstrap starting.
I20260812 08:02:17.031028 24039 tablet_bootstrap.cc:654] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.031654 24042 tablet_bootstrap.cc:654] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.032219 24039 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9: No bootstrap required, opened a new log
I20260812 08:02:17.032331 24039 ts_tablet_manager.cc:1403] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.000s
I20260812 08:02:17.032701 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:17.032855 24039 raft_consensus.cc:359] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.032989 24039 raft_consensus.cc:385] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.033025 24039 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.033169 24042 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: No bootstrap required, opened a new log
I20260812 08:02:17.033185 24039 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.033241 24042 ts_tablet_manager.cc:1403] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:02:17.033532 24039 ts_tablet_manager.cc:1434] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:02:17.033735 24042 raft_consensus.cc:359] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.033826 24042 raft_consensus.cc:385] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.033926 24042 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.034009 24039 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: Bootstrap starting.
W20260812 08:02:17.034010 23802 leader_election.cc:343] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:02:17.034206 24042 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.034490 24041 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:17.034566 24042 ts_tablet_manager.cc:1434] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20260812 08:02:17.034587 24041 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.034967 24041 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:17.035056 24039 tablet_bootstrap.cc:654] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.035485 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:17.033627 24060 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:17.035876 24060 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.036428 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:17.037019 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:17.037170 23450 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 0.
I20260812 08:02:17.037380 24042 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944: Bootstrap starting.
W20260812 08:02:17.037446 23802 leader_election.cc:343] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:02:17.037974 23807 leader_election.cc:343] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:02:17.038069 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:17.038395 24042 tablet_bootstrap.cc:654] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:17.038558 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 9c1d86a754e645269bc0796059331944
I20260812 08:02:17.039039 24041 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:02:17.039204 24041 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:17.039253 24041 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:17.039295 24041 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.040469 24041 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.040822 24041 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:17.041458 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:17.041519 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:17.041640 23450 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.042057 24060 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
W20260812 08:02:17.042639 23802 leader_election.cc:343] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:02:17.042795 23450 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 1.
I20260812 08:02:17.043154 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 9c1d86a754e645269bc0796059331944
I20260812 08:02:17.044181 24041 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:17.044273 24041 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:17.044576 24041 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.046325 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:17.046535 23686 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 0.
I20260812 08:02:17.046713 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:17.046896 23918 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 0.
I20260812 08:02:17.047369 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:17.047528 24039 tablet_bootstrap.cc:492] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: No bootstrap required, opened a new log
I20260812 08:02:17.047640 24039 ts_tablet_manager.cc:1403] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.000s
I20260812 08:02:17.047672 24042 tablet_bootstrap.cc:492] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944: No bootstrap required, opened a new log
I20260812 08:02:17.047770 24042 ts_tablet_manager.cc:1403] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.002s
I20260812 08:02:17.048220 24039 raft_consensus.cc:359] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.048282 24042 raft_consensus.cc:359] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.048349 24039 raft_consensus.cc:385] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.048378 24042 raft_consensus.cc:385] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:17.048395 24039 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.048413 24042 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Initialized, Role: FOLLOWER
I20260812 08:02:17.048522 24039 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.048601 24042 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.048794 24039 ts_tablet_manager.cc:1434] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20260812 08:02:17.048869 24042 ts_tablet_manager.cc:1434] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20260812 08:02:17.049001 23211 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 1 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:17.050006 24063 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:17.050115 24063 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:17.051194 24063 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:17.052121 24048 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:17.052220 24048 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:17.052282 24048 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.053292 24048 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.053902 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:17.054019 23686 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.054198 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:17.054317 23918 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.055181 23686 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 1.
I20260812 08:02:17.055348 23918 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 1.
I20260812 08:02:17.055668 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:17.055897 24060 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:17.055992 24060 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:17.056308 24060 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.056521 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:17.057183 24054 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:17.057291 24054 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.057840 24054 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:17.058583 24048 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 1 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:17.058701 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:17.058782 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:17.058868 23420 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 0.
I20260812 08:02:17.058887 23450 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 1: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:17.059171 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:17.059412 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:17.059492 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:17.059525 23918 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 0.
I20260812 08:02:17.059837 24054 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:17.059904 24054 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:17.059942 24054 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.059863 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 1 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:17.060822 24054 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.061208 24054 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:17.061324 24054 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.061553 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:17.061652 23450 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.062100 24054 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:02:17.062714 23450 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:17.063300 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:17.063627 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:17.063737 23918 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:17.064795 23918 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:17.065117 24063 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:17.065310 24063 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:17.065848 24063 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:17.069355 23210 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 1 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:17.071511 24066 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:17.076136 24066 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:17.079442 24048 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:17.083474 24055 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 08:02:17.086081 23540 tablet.cc:2406] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:17.091513 24063 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 08:02:17.633747 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:17.131520 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:58402)
0812 08:02:17.131661 (+   141us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:17.131677 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:17.131748 (+    71us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:17.633235 (+501487us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 08:02:17.634039 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
I20260812 08:02:18.126519 24066 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:18.126642 24066 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:18.126947 24089 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:18.127039 24066 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:18.127059 24089 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:18.128742 24101 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:18.128842 24101 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:18.129020 24054 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:18.129125 24054 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:18.129235 24101 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:18.129534 24054 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:18.129674 24089 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:18.131059 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:18.131186 23913 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:18.131768 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:18.133061 23684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:18.133186 23684 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 1.
I20260812 08:02:18.133482 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:18.133786 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:18.134054 24089 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:18.134114 24089 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:18.134150 24089 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:18.134181 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:18.134469 24054 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:18.134526 24054 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:18.134573 24054 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:18.135143 24089 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:18.135469 24089 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:18.135677 24054 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
W20260812 08:02:18.135928 24104 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:18.129331 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.22.155.3:44619)
0812 08:02:18.134825 (+  5494us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:18.134830 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:18.134848 (+    18us) server_negotiation.cc:408] Connection header received
0812 08:02:18.134925 (+    77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:18.134929 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:18.134989 (+    60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:18.135157 (+   168us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58402: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":5386,"thread_start_us":2698,"threads_started":1}
I20260812 08:02:18.136092 24054 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:18.136063 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:18.136587 23685 raft_consensus.cc:2393] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 2: Already voted for candidate 9c1d86a754e645269bc0796059331944 in this term.
I20260812 08:02:18.136575 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:18.136703 23911 raft_consensus.cc:2393] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 2: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
W20260812 08:02:18.145237 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:17.639319 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47002)
0812 08:02:17.639498 (+   179us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:17.639513 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:17.639595 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:18.145135 (+505540us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 08:02:18.153569 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36666: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:18.203263 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.075s (SENT)
W20260812 08:02:18.753386 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:18.223698 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47010)
0812 08:02:18.223882 (+   184us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:18.223900 (+    18us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:18.223994 (+    94us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:18.753264 (+529270us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 08:02:18.753628 24011 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
W20260812 08:02:19.224067 23570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.086s (SENT)
I20260812 08:02:19.227152 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.227998 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.228103 23449 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:19.228295 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:19.228500 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.228600 23441 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:19.228768 23440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:19.228982 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.229367 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.229447 23436 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 1.
I20260812 08:02:19.229600 23435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.229815 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:19.230028 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.230113 23431 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 1.
I20260812 08:02:19.230293 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.230530 24122 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:19.230602 24122 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:19.231069 24122 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:19.231448 24122 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:19.231510 24122 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
W20260812 08:02:19.231627 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36676: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
I20260812 08:02:19.231895 24122 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:19.232316 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:19.232501 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:19.232865 24101 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:19.233209 24101 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:19.233270 24101 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:19.233307 24101 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:19.234370 24101 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:19.234695 24101 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:19.234771 24101 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 08:02:19.235570 23686 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 1. Current term is 2. Ops: []
I20260812 08:02:19.235803 23684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:19.236577 24132 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:19.233376 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.22.155.1:46399)
0812 08:02:19.235715 (+  2339us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:19.235719 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:19.235733 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:19.235804 (+    71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:19.235807 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:19.235861 (+    54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:19.236006 (+   145us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47010: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2262,"thread_start_us":57,"threads_started":1}
I20260812 08:02:19.237015 24048 consensus_queue.cc:1059] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 08:02:19.237180 24048 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 LEADER]: Stepping down as leader of term 1
W20260812 08:02:19.237202 24132 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:19.235068 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.22.155.1:46399)
0812 08:02:19.236909 (+  1841us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:19.236913 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:19.236925 (+    12us) server_negotiation.cc:408] Connection header received
0812 08:02:19.236981 (+    56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:19.236984 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:19.237027 (+    43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:19.237120 (+    93us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47002: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":51,"threads_started":1}
I20260812 08:02:19.237223 24048 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:19.237360 24048 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:19.237598 24048 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:19.239080 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:19.239552 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:19.242316 24133 mvcc.cc:204] Tried to move back new op lower bound from 7317593039584628736 to 7317593035025108992. Current Snapshot: MvccSnapshot[applied={T|T < 7317593039584628736}]
W20260812 08:02:19.253203 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.272614 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.296106 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:19.308917 24060 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:19.309121 24060 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:19.309625 24060 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:19.310106 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:19.310245 23685 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 2.
I20260812 08:02:19.310627 23336 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:19.310832 24060 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:19.310891 24060 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:19.310920 24060 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:19.311782 24060 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:19.312233 24060 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:19.312621 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:19.312772 23685 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:19.313868 23685 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
I20260812 08:02:19.314244 23336 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
W20260812 08:02:19.314436 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:19.314445 24060 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:02:19.314554 24060 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:19.314694 24060 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:19.316419 23212 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:19.333627 23341 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.102s (SENT)
W20260812 08:02:19.334949 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:19.343919 23685 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 08:02:19.344391 24124 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 08:02:19.354605 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.384264 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.406881 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.407801 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.086s (SENT)
I20260812 08:02:19.407900 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:19.408077 24126 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 08:02:19.434540 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.436784 23341 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.102s (SENT)
I20260812 08:02:19.436919 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:19.437318 24124 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 08:02:19.451339 23341 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.109s (SENT)
I20260812 08:02:19.451472 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:19.451718 24060 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 08:02:19.459872 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.475517 23341 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.082s (SENT)
W20260812 08:02:19.484921 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.491976 23341 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.089s (SENT)
I20260812 08:02:19.498621 24126 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:19.498726 24126 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:19.499202 24126 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:19.499590 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:19.499734 23430 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 1.
I20260812 08:02:19.500037 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:19.500243 24126 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:19.500310 24126 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:19.500366 24126 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:19.501295 24126 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:19.501632 24126 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:19.502008 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:19.502136 23430 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:02:19.503002 23430 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 3.
I20260812 08:02:19.503319 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:19.503504 24126 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:02:19.503609 24126 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:19.503762 24126 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:19.505415 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 1 to 3, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 3 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } }
W20260812 08:02:19.508279 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:19.528739 23430 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260812 08:02:19.529224 24126 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 08:02:19.533775 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.554726 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.583326 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.607151 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.630482 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.655843 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.678026 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.702591 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.725267 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.727237 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.112s (SENT)
W20260812 08:02:19.743644 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.745304 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:19.236394 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:40924)
0812 08:02:19.240069 (+  3675us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:19.240085 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:19.240162 (+    77us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:19.745160 (+504998us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":3563}
W20260812 08:02:19.767150 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.769068 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.134s (SENT)
W20260812 08:02:19.789337 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.808555 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.828048 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.846298 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.865964 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.896176 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.915844 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.941066 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.968477 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:19.996515 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.024591 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.048300 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.066682 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.096131 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.116084 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.137456 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.165020 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.188482 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.218946 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.244511 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.275199 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.289309 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:19.763588 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:40940)
0812 08:02:19.763709 (+   121us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:19.763722 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:19.763791 (+    69us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:20.289184 (+525393us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 08:02:20.289508 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:20.294386 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:20.296726 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47018: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
I20260812 08:02:20.316922 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
W20260812 08:02:20.317263 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.075s (SENT)
I20260812 08:02:20.317339 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.317512 23906 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260812 08:02:20.317680 23918 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 1. Current term is 2. Ops: []
I20260812 08:02:20.317709 23904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.317793 23904 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 2.
I20260812 08:02:20.317967 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.318081 23909 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:20.319355 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.319455 23909 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
I20260812 08:02:20.319595 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.319720 23913 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260812 08:02:20.319855 23907 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260812 08:02:20.320106 23912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.320218 23912 raft_consensus.cc:2376] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3. Re-sending same reply.
I20260812 08:02:20.320446 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.320518 23903 raft_consensus.cc:2376] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3. Re-sending same reply.
W20260812 08:02:20.320628 24174 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:20.320000 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.22.155.3:44619)
0812 08:02:20.320244 (+   244us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:20.320249 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:20.320263 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:20.320307 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:20.320310 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:20.320359 (+    49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:20.320483 (+   124us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40924: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260812 08:02:20.317339 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:20.321043 24103 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 08:02:20.321106 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.080s (SENT)
W20260812 08:02:20.321689 24174 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:20.321265 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.22.155.3:44619)
0812 08:02:20.321415 (+   150us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:20.321419 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:20.321433 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:20.321472 (+    39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:20.321475 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:20.321519 (+    44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:20.321618 (+    99us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40940: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 08:02:20.321945 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.322134 23911 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 LEADER]: Stepping down as leader of term 1
I20260812 08:02:20.322202 23911 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:20.322393 23911 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:20.322654 23907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.322667 23911 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:02:20.323696 23912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.323858 23911 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260812 08:02:20.323956 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.324209 24054 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260812 08:02:20.324919 23918 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 3.
I20260812 08:02:20.325350 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:20.326418 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:20.326848 24183 mvcc.cc:204] Tried to move back new op lower bound from 7317593048283537408 to 7317593045008113664. Current Snapshot: MvccSnapshot[applied={T|T < 7317593048283537408}]
W20260812 08:02:20.865337 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:20.349758 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36718)
0812 08:02:20.349940 (+   182us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:20.349956 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:20.350036 (+    80us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:20.865189 (+515153us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 08:02:21.352072 24066 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:21.352113 24103 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:21.352197 24066 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
W20260812 08:02:21.352319 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:21.352207 24103 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:21.352624 24066 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:21.352762 24103 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:21.353363 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:21.353367 24126 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:21.353524 24126 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:21.353716 23684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:21.353850 24126 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
W20260812 08:02:21.354064 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.101s (SENT)
I20260812 08:02:21.354121 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:21.354466 24103 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 08:02:21.354977 24162 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:21.351181 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.22.155.2:33271)
0812 08:02:21.352894 (+  1713us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:21.352899 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:21.352918 (+    19us) server_negotiation.cc:408] Connection header received
0812 08:02:21.354208 (+  1290us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:21.354212 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:21.354271 (+    59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:21.354417 (+   146us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36718: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1618}
I20260812 08:02:21.355196 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:21.355312 23911 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 3.
I20260812 08:02:21.355700 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:21.355866 24054 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:02:21.355930 24054 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:21.355973 24054 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:21.357074 24054 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:21.357431 24054 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:21.357878 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:21.357986 23911 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:21.358973 23911 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 4.
I20260812 08:02:21.359229 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:21.359401 24054 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 08:02:21.359494 24054 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:21.359628 24054 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:21.361383 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 3 to 4, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 4 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:21.377229 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:20.870249 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47030)
0812 08:02:20.870463 (+   214us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:20.870479 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:20.870572 (+    93us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:21.377133 (+506561us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 08:02:21.377427 24011 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
W20260812 08:02:21.379866 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:21.383389 23911 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 08:02:21.383846 24126 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 08:02:21.385046 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40946: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
W20260812 08:02:21.400226 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:21.427706 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:21.953392 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:21.445300 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36768)
0812 08:02:21.445475 (+   175us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:21.445489 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:21.445571 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:21.953262 (+507691us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 08:02:22.449124 24225 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:22.446821 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.22.155.2:33271)
0812 08:02:22.447112 (+   291us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:22.447117 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:22.447135 (+    18us) server_negotiation.cc:408] Connection header received
0812 08:02:22.447192 (+    57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:22.447197 (+     5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:22.448738 (+  1541us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:22.448909 (+   171us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36768: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":61,"threads_started":1}
I20260812 08:02:22.449709 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:22.449981 23435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:22.450148 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:22.450191 24231 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:22.450284 24231 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:22.450716 24231 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:22.450742 24124 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 08:02:22.451027 23437 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:22.451131 23436 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 08:02:22.451185 23436 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:22.451229 23440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:22.451313 23436 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:22.451411 23436 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:22.451970 23435 raft_consensus.cc:1217] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.3]   Dedup: 1.3->[]
I20260812 08:02:22.452092 23435 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 1. Current term is 3. Ops: [1.2-1.3]
I20260812 08:02:22.451419 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:22.452473 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:22.452559 23436 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260812 08:02:22.452687 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:22.452806 23684 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 LEADER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 3. Current term is 4. Ops: []
I20260812 08:02:22.452843 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:22.456377 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47044: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:22.456617 24228 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:22.449914 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.22.155.1:46399)
0812 08:02:22.456264 (+  6350us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:22.456268 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:22.456280 (+    12us) server_negotiation.cc:408] Connection header received
0812 08:02:22.456326 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:22.456329 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:22.456372 (+    43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:22.456483 (+   111us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47030: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":6282,"thread_start_us":70,"threads_started":1}
I20260812 08:02:22.456597 24126 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260812 08:02:22.478682 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.508716 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.532138 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.552910 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.572546 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.593814 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.612152 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.636598 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.665931 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.677520 23341 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.111s (SENT)
I20260812 08:02:22.677654 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:22.677940 24235 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 08:02:22.691990 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.712592 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.731312 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.759313 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.780272 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.799846 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.822774 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.847129 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.875487 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.896785 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.924630 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.950357 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.972311 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:22.977356 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:22.460750 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:40956)
0812 08:02:22.461266 (+   516us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:22.461282 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:22.461361 (+    79us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:22.977218 (+515857us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":414}
W20260812 08:02:22.977523 24010 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:22.998447 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:23.008488 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:23.009140 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:23.009306 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
W20260812 08:02:23.009366 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.075s (SENT)
I20260812 08:02:23.009445 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:23.009552 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:23.009671 23911 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
I20260812 08:02:23.009776 23907 raft_consensus.cc:1217] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.3]   Dedup: 3.3->[]
I20260812 08:02:23.009918 23907 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 3. Current term is 4. Ops: [3.2-3.3]
I20260812 08:02:23.014184 24103 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:23.014313 24103 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:23.014599 24066 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 08:02:23.014739 24103 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:23.015386 24259 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:23.014629 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.22.155.3:44619)
0812 08:02:23.015024 (+   395us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:23.015029 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:23.015045 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:23.015091 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:23.015093 (+     2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:23.015136 (+    43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:23.015264 (+   128us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40956: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":79,"threads_started":1}
I20260812 08:02:23.015635 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:23.553357 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:23.041464 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36812)
0812 08:02:23.041642 (+   178us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:23.041658 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:23.041778 (+   120us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:23.553233 (+511455us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 08:02:24.042131 24235 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.042281 24235 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:24.042490 24275 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.042565 24275 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.042584 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:24.042755 24235 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:24.042862 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:24.042881 24275 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.042958 23436 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260812 08:02:24.042758 23430 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260812 08:02:24.044039 24275 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.044131 24275 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
W20260812 08:02:24.045265 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.087s (SENT) [suppressed 4 similar messages]
W20260812 08:02:24.045420 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.092s (SENT)
I20260812 08:02:24.045481 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:24.045746 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:24.045884 23913 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
I20260812 08:02:24.045899 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:24.045976 23909 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 1.
I20260812 08:02:24.046110 24066 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.046193 24066 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.046224 24103 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:02:24.046629 24066 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:24.047113 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.047289 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.047808 24275 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.047902 24275 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:02:24.047936 24275 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.047995 24275 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:24.048642 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:24.048763 23430 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 1.
I20260812 08:02:24.048988 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:24.049134 23436 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260812 08:02:24.049294 24275 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:24.049963 24275 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:24.050267 24275 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:24.050457 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.050666 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:24.050737 24066 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:24.050783 24066 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.050822 24066 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:24.050921 24103 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:02:24.051100 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:24.051198 23909 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 4.
W20260812 08:02:24.051357 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47056: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
I20260812 08:02:24.051707 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:24.051805 23913 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:24.051801 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.052024 24235 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:02:24.052076 24235 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.052032 24066 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.052115 24235 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:24.052423 24066 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:24.052874 23913 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 4.
I20260812 08:02:24.053136 24275 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:24.053143 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:24.053218 24235 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.053431 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.053251 24275 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:24.053678 24235 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.053577 24277 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 08:02:24.054009 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:24.054109 23913 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:24.054296 24277 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:24.054451 24277 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:24.054555 24275 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.054744 23436 raft_consensus.cc:2393] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 2: Already voted for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in this term.
I20260812 08:02:24.055092 23913 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 5.
I20260812 08:02:24.055349 24275 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.055502 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.055629 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:24.055754 23913 raft_consensus.cc:2393] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in current term 2: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:24.056389 24275 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 08:02:24.056576 24275 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:24.056783 24275 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.056975 23210 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 3 to 4, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 4 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:24.061653 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 4 to 5, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 5 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:24.072098 23913 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20260812 08:02:24.072677 24275 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 08:02:24.080300 23913 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260812 08:02:24.080739 24275 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260812 08:02:24.205076 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.077s (SENT)
W20260812 08:02:24.217521 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.086s (SENT)
W20260812 08:02:24.235025 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.089s (SENT)
I20260812 08:02:24.235163 23336 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:24.235493 24235 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 08:02:24.239210 23802 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.095s (SENT)
W20260812 08:02:24.243714 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.099s (SENT)
W20260812 08:02:24.244002 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.099s (SENT)
W20260812 08:02:24.265362 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.105s (SENT)
W20260812 08:02:24.265682 23802 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.106s (SENT)
I20260812 08:02:24.265766 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:24.266009 24066 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 08:02:24.327250 24275 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:24.327382 24275 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.327740 24275 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.328071 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:24.328213 23913 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 2.
I20260812 08:02:24.328534 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.328717 24275 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:24.328814 24275 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:24.328876 24275 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:24.329818 24275 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.330156 24275 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:24.330499 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:24.330631 23913 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:24.331423 23913 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 3.
I20260812 08:02:24.331714 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:24.331923 24275 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:02:24.331995 24275 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:24.332149 24275 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.333802 23212 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 1 to 3, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 3 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:24.360539 23913 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 08:02:24.360988 24275 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 08:02:24.428864 23336 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.100s (SENT) [suppressed 10 similar messages]
W20260812 08:02:24.530216 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.100s (SENT)
W20260812 08:02:24.563719 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.116s (SENT)
W20260812 08:02:24.609268 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:24.087304 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36824)
0812 08:02:24.088456 (+  1152us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:24.088474 (+    18us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:24.088569 (+    95us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:24.609165 (+520596us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":1030}
W20260812 08:02:24.609549 24009 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:24.621752 23683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.622351 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.622567 23647 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 08:02:24.622629 23647 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:24.622716 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.622812 23647 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.622890 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
W20260812 08:02:24.622973 23570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:24.623085 23647 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:02:24.623162 23648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.623584 23649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.623806 23677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.623989 23679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.624187 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.624348 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.625010 23436 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 4. Current term is 5. Ops: []
I20260812 08:02:24.622637 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.625332 23430 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 3. Current term is 4. Ops: []
W20260812 08:02:24.625481 24248 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:24.622795 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.22.155.2:33271)
0812 08:02:24.625086 (+  2291us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:24.625091 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:24.625107 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:24.625152 (+    45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:24.625155 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:24.625206 (+    51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:24.625337 (+   131us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36812: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2208}
W20260812 08:02:24.626024 24248 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:24.623178 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.22.155.2:33271)
0812 08:02:24.625671 (+  2493us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:24.625677 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:24.625694 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:24.625738 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:24.625743 (+     5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:24.625804 (+    61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:24.625917 (+   113us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36824: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2406,"thread_start_us":100,"threads_started":1}
I20260812 08:02:24.625988 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.626137 23654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.626452 23668 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 08:02:24.626534 23668 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:24.626711 23668 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:24.626840 23668 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:24.626910 23657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.627214 23658 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 08:02:24.627269 23658 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:24.627436 23658 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:24.627573 23658 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:24.628042 23668 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20260812 08:02:24.628607 23684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.628603 23658 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260812 08:02:24.628710 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.628739 23684 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 4.
I20260812 08:02:24.628989 23680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.629267 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.629526 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:24.630476 23672 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 5.
I20260812 08:02:24.630940 23683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:24.631037 23683 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 4.
I20260812 08:02:24.631721 23909 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 3. Current term is 4. Ops: []
I20260812 08:02:24.624352 23647 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260812 08:02:25.185333 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:24.653836 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47086)
0812 08:02:24.653989 (+   153us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:24.654004 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:24.654089 (+    85us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:25.185215 (+531126us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 08:02:25.654623 24277 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 08:02:25.654955 24277 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260812 08:02:25.655404 24306 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:25.654590 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.22.155.1:46399)
0812 08:02:25.654900 (+   310us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:25.654906 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:25.654926 (+    20us) server_negotiation.cc:408] Connection header received
0812 08:02:25.654983 (+    57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:25.654988 (+     5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:25.655144 (+   156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:25.655282 (+   138us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47086: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 08:02:25.657665 24066 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.657783 24066 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:25.658195 24066 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:25.658267 24066 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.658318 24066 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:25.658597 24066 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:25.658658 24066 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.658699 24066 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:25.658926 24066 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:25.659866 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:25.659866 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:25.660241 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:25.697319 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:25.189217 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36834)
0812 08:02:25.189437 (+   220us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:25.189458 (+    21us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:25.189580 (+   122us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:25.697206 (+507626us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 08:02:25.697562 24009 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
W20260812 08:02:25.707285 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40970: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
W20260812 08:02:25.738705 23567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:25.738827 23567 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:25.739437 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:25.739596 23647 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 5.
W20260812 08:02:25.739626 23568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:25.741230 24336 mvcc.cc:204] Tried to move back new op lower bound from 7317593063714603008 to 7317593063648256000. Current Snapshot: MvccSnapshot[applied={T|T < 7317593063714603008}]
I20260812 08:02:25.741515 24349 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:25.741660 24349 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:25.741827 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:25.742084 23683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:25.742189 23683 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 3.
I20260812 08:02:25.742489 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:25.742498 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:25.742712 24349 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:25.742782 24349 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.742834 24349 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:25.742944 23568 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:25.743109 24349 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:25.743251 24349 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:02:25.743415 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:25.743431 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:25.743877 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:25.744011 24349 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:02:25.745296 24318 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 08:02:25.746285 24315 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:25.745130 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.22.155.2:33271)
0812 08:02:25.745959 (+   829us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:25.745964 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:25.745978 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:25.746022 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:25.746025 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:25.746073 (+    48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:25.746203 (+   130us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36834: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":731}
I20260812 08:02:25.747175 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9
I20260812 08:02:25.747378 24320 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 08:02:25.747437 24320 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.747478 24320 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:25.748541 24320 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:25.748930 24320 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:25.747488 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:25.749169 24320 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:02:25.749239 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9
I20260812 08:02:25.749367 24320 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:02:25.749437 24320 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:25.749500 24320 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:25.750547 24320 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:25.751174 24320 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:26.273463 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:25.772263 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47140)
0812 08:02:25.772455 (+   192us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:25.772470 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:25.772550 (+    80us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:26.273343 (+500793us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 08:02:26.777436 24352 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:26.777626 24352 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:26.778136 24352 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:26.778409 24349 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:26.778507 24349 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:26.778666 23683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:26.778796 23683 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:26.778887 24349 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:26.779074 24260 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:26.779145 24260 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:26.779299 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:26.779386 23647 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:26.779508 24260 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:26.779872 23683 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 6.
I20260812 08:02:26.780102 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:26.780278 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:26.780377 23656 raft_consensus.cc:2376] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 6. Re-sending same reply.
I20260812 08:02:26.780424 23647 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 4.
I20260812 08:02:26.780553 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:26.780625 23672 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 4.
I20260812 08:02:26.780959 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:26.780974 24349 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:26.781052 24349 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:26.781108 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:26.781122 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:26.781328 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:26.781412 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 4.
I20260812 08:02:26.781467 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:26.781499 24349 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:26.781262 23913 raft_consensus.cc:2393] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 4: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:26.781752 24352 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 08:02:26.781976 24352 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:26.782197 24352 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:26.782334 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:26.782624 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:26.782941 24349 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:02:26.782977 24352 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader election won for term 4
I20260812 08:02:26.783005 24349 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:26.783052 24349 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:26.783043 24352 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:26.783165 24352 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:26.783736 24352 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:02:26.783800 24352 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:26.783840 24352 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:26.784072 24349 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:26.784451 24349 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:26.784910 24352 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
W20260812 08:02:26.785312 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:26.279376 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:36844)
0812 08:02:26.279556 (+   180us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:26.279571 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:26.279659 (+    88us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:26.785177 (+505518us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 08:02:26.785485 24352 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:26.785538 24009 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:26.785642 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:26.785756 23672 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 5: Already voted for candidate 9c1d86a754e645269bc0796059331944 in this term.
I20260812 08:02:26.786269 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:26.786403 23913 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 5: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:26.786829 23212 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 5 to 6, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 6 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
W20260812 08:02:26.788208 24354 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:26.778058 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.22.155.2:33271)
0812 08:02:26.787841 (+  9783us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:26.787846 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:26.787862 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:26.787908 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:26.787912 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:26.787964 (+    52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:26.788104 (+   140us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36844: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":9700,"thread_start_us":66,"threads_started":1}
I20260812 08:02:26.791002 23212 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 3 to 4, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 4 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:26.795450 23672 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 5 index: 9. Preceding OpId from leader: term: 6 index: 14. (index mismatch)
I20260812 08:02:26.795994 24360 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.000s
I20260812 08:02:26.809424 23672 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20260812 08:02:26.809983 24360 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 08:02:26.856513 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.080s (SENT)
W20260812 08:02:26.863981 23570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.084s (SENT) [suppressed 7 similar messages]
W20260812 08:02:26.867525 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.091s (SENT)
W20260812 08:02:26.942382 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.080s (SENT)
W20260812 08:02:26.949728 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.084s (SENT)
W20260812 08:02:26.985138 23570 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.101s (SENT)
I20260812 08:02:26.985234 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:26.985409 24349 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 08:02:27.010677 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.112s (SENT)
I20260812 08:02:27.010954 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:27.011191 24349 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260812 08:02:27.029706 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.121s (SENT)
I20260812 08:02:27.029815 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:27.030019 24361 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260812 08:02:27.073392 23570 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.146s (SENT)
I20260812 08:02:27.073503 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:27.073685 24349 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:02:27.102070 24349 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:27.102231 24349 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:27.102785 24349 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:27.103297 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:27.103449 23913 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 5.
I20260812 08:02:27.103747 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:27.103962 24349 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 08:02:27.104049 24349 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:27.104103 24349 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:27.105046 24349 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:27.105381 24349 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:27.105733 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:27.105854 23913 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:27.106911 23913 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 6.
I20260812 08:02:27.107208 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:27.107414 24349 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 08:02:27.107506 24349 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:27.107635 24349 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:27.110230 23210 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 4 to 6, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 6 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:27.137310 23913 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260812 08:02:27.137913 24349 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260812 08:02:27.329242 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:26.800908 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:47152)
0812 08:02:26.801260 (+   352us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:26.801277 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:26.801367 (+    90us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:27.329142 (+527775us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":121}
W20260812 08:02:27.336589 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37834: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
W20260812 08:02:27.381067 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.137s (SENT)
W20260812 08:02:27.498396 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.197s (SENT)
W20260812 08:02:27.877267 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:27.349894 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48346)
0812 08:02:27.350021 (+   127us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:27.350035 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:27.350102 (+    67us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:27.877155 (+527053us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 08:02:27.877559 24011 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
I20260812 08:02:27.891853 23672 raft_consensus.cc:1217] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 5.9->[5.10-5.12]   Dedup: 5.12->[]
I20260812 08:02:27.891911 23647 raft_consensus.cc:1217] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[3.6-3.7]   Dedup: 3.7->[]
I20260812 08:02:27.892042 23672 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 5. Current term is 6. Ops: [5.10-5.12]
I20260812 08:02:27.892048 23647 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 3. Current term is 4. Ops: [3.6-3.7]
W20260812 08:02:28.449337 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:27.923274 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37842)
0812 08:02:27.923468 (+   194us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:27.923482 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:27.923560 (+    78us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:28.449227 (+525667us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 08:02:28.923930 24424 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:28.924088 24424 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:28.925443 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.925782 23437 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.925904 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.926028 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.926236 23440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.926461 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.926488 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.926725 23449 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 LEADER]: Stepping down as leader of term 4
I20260812 08:02:28.926777 23449 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:28.926921 23449 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:28.926986 24426 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:28.927042 23449 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Advancing to term 6
I20260812 08:02:28.927176 24235 consensus_queue.cc:1059] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 5.9, Next index: 10, Last known committed idx: 14, Time since last communication: 0.000s
I20260812 08:02:28.927323 24235 consensus_queue.cc:1059] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 1.5, Next index: 6, Last known committed idx: 9, Time since last communication: 0.000s
I20260812 08:02:28.927088 24426 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:28.927479 24428 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 08:02:28.927542 24428 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:28.927788 24426 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:28.927662 24428 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:28.927968 24428 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:28.928089 24424 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:28.928284 23449 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260812 08:02:28.928608 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.929095 24424 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 08:02:28.929579 23450 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 6.
I20260812 08:02:28.929625 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 6 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.929728 23420 raft_consensus.cc:2376] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 6. Re-sending same reply.
I20260812 08:02:28.929939 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.930140 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
W20260812 08:02:28.930301 23342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:28.930382 23435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.930578 24318 consensus_queue.cc:765] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 14 is ahead of the local log (next sequential op: 13). Destination peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 6.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260812 08:02:28.930897 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
W20260812 08:02:28.930932 23440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.155.3:39087 (request call id 25) took 1039 ms (client timeout 80 ms). Trace:
W20260812 08:02:28.930989 23440 rpcz_store.cc:269] 0812 08:02:28.929658 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.930797 (+  1139us) service_pool.cc:211] Skipping call since client already timed out
0812 08:02:28.930921 (+   124us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260812 08:02:28.931154 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.931303 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.931391 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.931576 23437 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.931891 23438 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 4.
I20260812 08:02:28.932183 23436 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20260812 08:02:28.932525 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 4 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.932610 23450 raft_consensus.cc:2376] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 4. Re-sending same reply.
I20260812 08:02:28.932917 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.933036 24275 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 08:02:28.933099 24275 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:28.933214 24275 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:28.933380 24275 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Advancing to term 6
W20260812 08:02:28.933531 24432 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:28.930255 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.22.155.1:46399)
0812 08:02:28.933128 (+  2873us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:28.933134 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:28.933153 (+    19us) server_negotiation.cc:408] Connection header received
0812 08:02:28.933199 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:28.933203 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:28.933258 (+    55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:28.933390 (+   132us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48346: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2636,"thread_start_us":59,"threads_started":1}
I20260812 08:02:28.933761 23429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:28.934257 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:28.934315 23647 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 LEADER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 4. Current term is 6. Ops: []
I20260812 08:02:28.934342 23431 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 6 index: 14. (index mismatch)
W20260812 08:02:28.934835 24434 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:28.931880 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.22.155.1:46399)
0812 08:02:28.934093 (+  2213us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:28.934099 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:28.934116 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:28.934158 (+    42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:28.934509 (+   351us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:28.934563 (+    54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:28.934689 (+   126us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47152: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1788,"thread_start_us":59,"threads_started":1}
W20260812 08:02:28.935290 24432 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:28.932301 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.22.155.1:46399)
0812 08:02:28.934976 (+  2675us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:28.934982 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:28.934997 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:28.935038 (+    41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:28.935041 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:28.935089 (+    48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:28.935190 (+   101us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47140: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":109,"threads_started":1}
W20260812 08:02:28.944866 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.948179 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.953373 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.964104 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.965214 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:28.458712 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37856)
0812 08:02:28.458854 (+   142us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:28.458867 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:28.458935 (+    68us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:28.965122 (+506187us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 08:02:28.965356 24010 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:28.972157 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:48360: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:28.976649 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.979892 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.980983 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54054: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:28.984505 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:28.997562 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.008921 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.010800 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.017167 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.023476 24275 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:29.023634 24275 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.024102 24275 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:29.024535 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:29.024755 23647 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 7 because replica has last-logged OpId of term: 6 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 12.
W20260812 08:02:29.029316 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.035643 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.036921 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.042161 24275 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:29.042464 24275 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.043167 24275 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:29.043700 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:29.043908 23647 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 5 because replica has last-logged OpId of term: 4 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 7.
W20260812 08:02:29.057991 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.058154 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.058928 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.079653 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.081983 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.087553 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.101521 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.101531 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.083s (SENT)
I20260812 08:02:29.101658 23568 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:29.101902 24424 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 08:02:29.106798 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.109493 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.122192 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.123097 23568 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.094s (SENT)
I20260812 08:02:29.123175 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:29.123364 24424 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 08:02:29.129631 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.132776 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.141280 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.152273 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.155200 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.168617 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.173959 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.176194 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.190995 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.193397 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.204758 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.212594 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.214089 24424 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:29.214201 24424 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.214690 24424 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
W20260812 08:02:29.214859 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.215178 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:29.215330 23447 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 4.
I20260812 08:02:29.215704 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:29.215946 24424 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:02:29.216027 24424 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:29.216089 24424 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:29.217156 24424 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.217561 24424 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:29.217935 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:29.218070 23447 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:29.218911 23447 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 5.
I20260812 08:02:29.219216 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:29.219393 24424 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 08:02:29.219506 24424 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:29.219660 24424 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 9, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.221288 23212 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 4 to 5, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 5 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:29.225489 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.229660 23341 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.102s (SENT)
I20260812 08:02:29.229734 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:29.229936 24275 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 08:02:29.231160 24426 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:29.231271 24426 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.231719 24426 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:29.232038 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:29.232187 23431 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 6.
I20260812 08:02:29.232475 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:29.232645 24426 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 08:02:29.232708 24426 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:29.232753 24426 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Advancing to term 7
W20260812 08:02:29.233230 23341 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.094s (SENT)
I20260812 08:02:29.233295 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:29.233479 24275 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:02:29.233830 24426 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.234202 24426 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:29.234489 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:29.234601 23431 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Advancing to term 7
W20260812 08:02:29.234618 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.235458 23431 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 7.
I20260812 08:02:29.235770 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:29.235944 24426 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 08:02:29.236028 24426 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:29.236171 24426 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 6.14, Last appended by leader: 14, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:29.237789 23212 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 6 to 7, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 7 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:29.243867 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.246992 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.247722 23431 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
I20260812 08:02:29.248175 24424 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
W20260812 08:02:29.261948 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:29.262192 23431 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 7 index: 15. (index mismatch)
I20260812 08:02:29.262549 24426 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 08:02:29.268975 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.271647 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.283344 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.289717 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.302835 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.309777 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.312031 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.324333 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.336609 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.339892 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.352468 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.360993 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.361104 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.377882 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.383457 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.389678 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.403095 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.405124 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.409536 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.419801 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.100s (SENT)
W20260812 08:02:29.422232 23568 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.093s (SENT)
W20260812 08:02:29.423938 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.428778 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.430918 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.452356 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.453081 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.453171 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.475467 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.475790 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.483785 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.484784 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.134s (SENT)
W20260812 08:02:29.497273 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.505354 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:28.992303 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37860)
0812 08:02:28.992462 (+   159us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:28.992475 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:28.992557 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:29.505215 (+512658us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 08:02:29.507468 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.508837 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.519948 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:48376: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:29.521579 23568 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.144s (SENT)
W20260812 08:02:29.525492 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.533841 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.538368 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.547286 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.553639 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.556964 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.576716 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.579946 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.580996 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.599771 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.599927 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.601126 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.623469 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.628649 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.631798 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.651600 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.653047 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.654263 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.670616 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.675565 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.679791 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.695875 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.701287 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.707316 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.722713 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.729800 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.734100 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.746531 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.754132 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.759434 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.768821 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.776369 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.781831 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.788051 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.800220 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.802461 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.811314 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.823841 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.830900 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.836153 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.843034 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.852531 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.866305 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.873499 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.876725 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.893941 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.898864 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.906332 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.920358 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.923149 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.929695 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.944298 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.946058 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.953675 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.973201 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.974023 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.978454 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.996789 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.996896 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:29.998216 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.021828 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.025066 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.026597 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.048929 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.049327 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:29.537756 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37864)
0812 08:02:29.537842 (+    86us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:29.537854 (+    12us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:29.537917 (+    63us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:30.049226 (+511309us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 08:02:30.049556 24010 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:30.050143 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.050257 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:30.070813 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:30.074436 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.074854 23889 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 08:02:30.074934 23889 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:30.075068 23912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.075143 23886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:30.075451 23901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.075500 23907 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 LEADER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 3. Current term is 4. Ops: []
I20260812 08:02:30.075616 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.075743 23887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:30.075793 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.075943 23913 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 4. Current term is 6. Ops: []
I20260812 08:02:30.076011 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076081 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076294 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076395 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076555 23902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076733 23902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.076862 23902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:30.077005 24361 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:30.077085 24361 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:30.077255 23892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:30.077088 23893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 5 candidate_status { last_received { term: 4 index: 9 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:30.077502 24361 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:30.077793 24404 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 08:02:30.077930 23890 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 LEADER]: Stepping down as leader of term 4
I20260812 08:02:30.077988 23890 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:30.078130 23890 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.9, Last appended by leader: 9, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:30.078281 23890 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:30.078334 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:30.075071 23889 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 6.14, Last appended by leader: 14, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
W20260812 08:02:30.079432 24484 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:30.076259 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.22.155.3:44619)
0812 08:02:30.078309 (+  2050us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:30.078319 (+    10us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:30.078337 (+    18us) server_negotiation.cc:408] Connection header received
0812 08:02:30.078397 (+    60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:30.078402 (+     5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:30.078460 (+    58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:30.078594 (+   134us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37856: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":59,"threads_started":1}
I20260812 08:02:30.079636 23890 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 4 index: 9. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
I20260812 08:02:30.079619 24404 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 08:02:30.079960 24484 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:30.076916 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.22.155.3:44619)
0812 08:02:30.079645 (+  2729us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:30.079651 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:30.079668 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:30.079708 (+    40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:30.079711 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:30.079759 (+    48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:30.079865 (+   106us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37860: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2665,"thread_start_us":71,"threads_started":1}
I20260812 08:02:30.080076 23889 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Advancing to term 7
W20260812 08:02:30.080418 24484 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:30.077724 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.22.155.3:44619)
0812 08:02:30.080103 (+  2379us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:30.080107 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:30.080120 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:30.080155 (+    35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:30.080158 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:30.080199 (+    41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:30.080317 (+   118us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2328,"thread_start_us":90,"threads_started":1}
W20260812 08:02:30.080925 24484 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:30.077877 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.22.155.3:44619)
0812 08:02:30.080599 (+  2722us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:30.080603 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:30.080615 (+    12us) server_negotiation.cc:408] Connection header received
0812 08:02:30.080653 (+    38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:30.080656 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:30.080713 (+    57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:30.080838 (+   125us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2672,"thread_start_us":93,"threads_started":1}
I20260812 08:02:30.081283 23889 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 6 index: 14. Preceding OpId from leader: term: 7 index: 15. (index mismatch)
I20260812 08:02:30.081604 23909 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 5. Current term is 7. Ops: []
I20260812 08:02:30.082301 24426 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260812 08:02:30.082372 24492 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260812 08:02:30.625329 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:30.106254 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:54096)
0812 08:02:30.106429 (+   175us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:30.106443 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:30.106522 (+    79us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:30.625215 (+518693us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 08:02:31.106248 23334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:31.106972 24504 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
W20260812 08:02:31.107034 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:31.107107 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:31.107071 24504 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.107509 24504 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.107587 24504 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.107643 24504 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.107810 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:31.107916 23447 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 6.
I20260812 08:02:31.108011 24504 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:31.108083 24504 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.108136 24504 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.108242 23431 raft_consensus.cc:1217] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.7->[4.8-4.9]   Dedup: 4.9->[]
I20260812 08:02:31.108311 23431 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 4. Current term is 5. Ops: [4.8-4.9]
W20260812 08:02:31.108881 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.102s (SENT) [suppressed 8 similar messages]
I20260812 08:02:31.109457 24480 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.109560 24480 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.110050 24480 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.110829 24504 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
W20260812 08:02:31.111689 23801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:31.113540 24480 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:31.113585 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:31.113629 24480 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.113685 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 6.
I20260812 08:02:31.113958 24480 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.114346 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:31.114477 23431 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 6.
I20260812 08:02:31.114684 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:31.114807 23909 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 7.
I20260812 08:02:31.114982 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:31.115131 23431 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 5.
I20260812 08:02:31.115574 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:31.115613 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 9c1d86a754e645269bc0796059331944
I20260812 08:02:31.115681 23909 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 5.
I20260812 08:02:31.115823 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:31.115847 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.115986 23431 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 8 because replica has last-logged OpId of term: 7 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 14.
I20260812 08:02:31.115999 24480 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 08:02:31.116043 24480 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.116104 24480 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:02:31.115608 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.116377 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.116619 24505 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 08:02:31.116633 24504 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 08:02:31.116725 24505 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.116758 24504 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.116880 24505 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.116928 24504 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:02:31.117328 24480 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.117662 24480 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 7 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.117771 24480 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 08:02:31.117820 24480 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:31.117873 24480 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:31.118252 24505 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.118387 24504 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.118747 24505 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:31.118806 24480 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.119062 24504 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.119135 24480 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.119704 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:31.119727 23889 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:31.119808 23889 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.119849 23431 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 7: Already voted for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in this term.
I20260812 08:02:31.120044 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:31.120124 23447 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:31.120703 23889 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 8.
I20260812 08:02:31.120930 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.121093 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:31.121217 23909 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in current term 7: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:31.121280 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.121691 23447 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 6.
I20260812 08:02:31.122053 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.122218 24487 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Leader election won for term 6
I20260812 08:02:31.122293 24487 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:31.122427 24487 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.122579 24505 raft_consensus.cc:2764] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260812 08:02:31.122707 24505 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 08:02:31.122785 24505 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:31.122896 24505 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 7.16, Last appended by leader: 16, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.124083 23210 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 5 to 6, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 6 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:31.125005 23212 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 7 to 8, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 8 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:31.126075 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37870: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:31.152745 23909 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 6 index: 14. Preceding OpId from leader: term: 8 index: 17. (index mismatch)
I20260812 08:02:31.153928 23447 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20260812 08:02:31.153857 24505 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260812 08:02:31.154459 24487 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 08:02:31.187299 23336 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.079s (SENT) [suppressed 3 similar messages]
W20260812 08:02:31.267464 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.079s (SENT)
W20260812 08:02:31.280838 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.080s (SENT)
I20260812 08:02:31.281093 23336 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:31.281316 24507 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 08:02:31.283345 23802 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.086s (SENT)
W20260812 08:02:31.319589 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.103s (SENT)
W20260812 08:02:31.321913 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.101s (SENT)
W20260812 08:02:31.322085 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.104s (SENT)
W20260812 08:02:31.330588 23802 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.105s (SENT)
I20260812 08:02:31.330675 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:31.330937 24487 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260812 08:02:31.335228 23802 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.110s (SENT)
I20260812 08:02:31.335297 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:31.335495 24487 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 08:02:31.340461 23802 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.110s (SENT)
I20260812 08:02:31.392581 24507 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:31.392779 24507 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.393261 24507 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.393610 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:31.393764 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 7.
I20260812 08:02:31.394115 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.394299 24507 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 08:02:31.394384 24507 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:31.394460 24507 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.395380 24507 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.395752 24507 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:31.396091 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:31.396195 23909 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.397014 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 8.
I20260812 08:02:31.397396 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:31.397603 24507 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 08:02:31.397720 24507 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:31.397898 24507 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.10, Last appended by leader: 10, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.399679 23210 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 6 to 8, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 8 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:31.418606 23909 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 6 index: 10. Preceding OpId from leader: term: 8 index: 11. (index mismatch)
I20260812 08:02:31.419126 24507 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260812 08:02:31.570824 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.088s (SENT)
W20260812 08:02:31.649304 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:31.135072 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:54104)
0812 08:02:31.135201 (+   129us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:31.135215 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:31.135273 (+    58us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:31.649204 (+513931us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 08:02:31.649520 24009 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
W20260812 08:02:31.660135 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37880: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:31.692540 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.692739 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.692958 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693014 23648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.693173 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.693207 23657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
W20260812 08:02:31.693343 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.148s (SENT)
I20260812 08:02:31.693466 23654 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693542 23677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693567 23679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693485 23656 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 LEADER]: Stepping down as leader of term 6
I20260812 08:02:31.693665 23656 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:31.693738 23670 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 08:02:31.693812 23656 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 6.10, Last appended by leader: 10, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:31.693888 23672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693910 23658 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 08:02:31.693951 23658 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:31.693981 23656 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.694080 23683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.694181 23684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 6 index: 14 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.694386 23647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.694595 23686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.694788 23649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.694998 23680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.695073 23656 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 6 index: 10. Preceding OpId from leader: term: 8 index: 11. (index mismatch)
I20260812 08:02:31.695169 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.693181 23648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.695253 23668 raft_consensus.cc:2368] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for earlier term 7. Current term is 8.
I20260812 08:02:31.692965 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.695370 23655 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 8.
I20260812 08:02:31.695385 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.693475 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:31.695585 23674 raft_consensus.cc:2368] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 7. Current term is 8.
I20260812 08:02:31.696661 23447 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 7. Current term is 8. Ops: []
W20260812 08:02:31.696852 24544 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:31.696012 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.22.155.2:33271)
0812 08:02:31.696238 (+   226us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:31.696246 (+     8us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:31.696262 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:31.696325 (+    63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:31.696329 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:31.696390 (+    61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:31.696687 (+   297us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54096: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260812 08:02:31.696957 23431 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 5. Current term is 6. Ops: []
I20260812 08:02:31.693804 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:31.693804 23670 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:31.698175 23670 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.698346 23670 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:02:31.698388 23909 raft_consensus.cc:1217] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Deduplicated request from leader. Original: 6.14->[7.15-7.16]   Dedup: 7.16->[]
I20260812 08:02:31.698511 23909 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 7. Current term is 8. Ops: [7.15-7.16]
I20260812 08:02:31.699370 23670 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 6 index: 13. (index mismatch)
I20260812 08:02:31.694068 23658 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 7.16, Last appended by leader: 16, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:31.699749 23658 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:31.699828 24487 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 08:02:31.699749 24492 consensus_queue.cc:1175] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" responder_term: 8 status { last_received { term: 8 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260812 08:02:31.700803 23658 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 7 index: 16. Preceding OpId from leader: term: 8 index: 17. (index mismatch)
W20260812 08:02:31.701686 24544 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:31.701235 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.22.155.2:33271)
0812 08:02:31.701358 (+   123us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:31.701365 (+     7us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:31.701382 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:31.701428 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:31.701431 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:31.701488 (+    57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:31.701600 (+   112us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54104: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":31}
W20260812 08:02:32.225353 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:31.724565 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48416)
0812 08:02:31.724741 (+   176us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:31.724759 (+    18us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:31.724849 (+    90us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:32.225228 (+500379us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 08:02:31.696662 23420 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 6. Current term is 8. Ops: []
I20260812 08:02:32.725214 23420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.155.2:39949 (request call id 327) took 1028 ms. Trace:
I20260812 08:02:32.725268 23420 rpcz_store.cc:276] 0812 08:02:31.696507 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.696557 (+    50us) service_pool.cc:224] Handling call
0812 08:02:31.696628 (+    71us) raft_consensus.cc:1471] Updating replica for 0 ops
0812 08:02:32.725130 (+1028502us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.725202 (+    72us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260812 08:02:32.725441 24505 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260812 08:02:32.725835 24505 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260812 08:02:32.727435 24560 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:32.727625 24560 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:32.728303 24560 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:32.729689 24487 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:32.729770 24487 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:32.730086 24487 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:32.730787 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:32.731365 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:32.731551 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:32.227246 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37890)
0812 08:02:32.227482 (+   236us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:32.227502 (+    20us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:32.227599 (+    97us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:32.731453 (+503854us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 25 requested bytes
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 08:02:32.731734 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 25 requested bytes
W20260812 08:02:32.731868 24563 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:32.729316 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.22.155.3:44619)
0812 08:02:32.731165 (+  1849us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:32.731169 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:32.731184 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:32.731228 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:32.731231 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:32.731279 (+    48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:32.731748 (+   469us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37890: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":54,"threads_started":1}
W20260812 08:02:32.732796 24537 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:32.727576 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.22.155.1:46399)
0812 08:02:32.732473 (+  4897us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:32.732477 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:32.732491 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:32.732530 (+    39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:32.732533 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:32.732579 (+    46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:32.732683 (+   104us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48416: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":4818}
W20260812 08:02:32.764698 24548 log.cc:927] Time spent T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944: Append to log took a long time: real 1.063s	user 0.001s	sys 0.000s
W20260812 08:02:32.763491 23566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:32.763656 23568 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:32.765717 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:32.763954 24426 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:32.765866 24426 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:32.766012 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:32.766268 24426 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:32.764146 24492 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:32.766387 24492 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:32.766662 24492 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:32.764081 24427 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:32.767066 24427 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:32.767657 24427 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:32.768955 23674 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 8.
I20260812 08:02:32.769461 23889 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:32.769776 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:32.770386 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:32.770833 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:32.771076 24561 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 08:02:32.771169 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9
I20260812 08:02:32.771348 24561 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 08:02:32.771394 24561 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:32.771445 24561 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:32.772508 24561 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:32.772868 24561 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:32.774055 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
W20260812 08:02:33.313319 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:32.795343 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48466)
0812 08:02:32.795564 (+   221us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:32.795580 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:32.795667 (+    87us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:33.313211 (+517544us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 08:02:33.796005 24561 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:33.796116 24561 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:33.796545 24561 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:33.797858 23570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.078s (SENT) [suppressed 11 similar messages]
I20260812 08:02:33.798602 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:33.804044 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54114: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
I20260812 08:02:33.804728 23670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.155.1:47399 (request call id 97) took 1039 ms. Trace:
I20260812 08:02:33.804816 23670 rpcz_store.cc:276] 0812 08:02:32.765418 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.765592 (+   174us) service_pool.cc:224] Handling call
0812 08:02:32.765721 (+   129us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:32.766852 (+  1131us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:32.766855 (+     3us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0812 08:02:32.766970 (+   115us) write_op.cc:270] Start()
0812 08:02:32.766992 (+    22us) write_op.cc:276] Timestamp: P: 1786521751673156 usec, L: 0
0812 08:02:32.767215 (+   223us) write_op.cc:270] Start()
0812 08:02:32.767229 (+    14us) write_op.cc:276] Timestamp: P: 1786521751688290 usec, L: 0
0812 08:02:32.767404 (+   175us) log.cc:844] Serialized 618 byte log entry
0812 08:02:32.767513 (+   109us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:02:32.767549 (+    36us) log.cc:844] Serialized 14 byte log entry
0812 08:02:32.767618 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.767864 (+   246us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x564d957b0794. stack: 00007f6d4f57fd35 00007f6d4f580325 00007f6d4f08c82d 0000564d6a578b18 0000564d6a578b6b 0000564d6a57c9f2 00007f6d515d54af 00007f6d515d0941 00007f6d526c538c 00007f6d50d29599 00007f6d50d2dbc4 00007f6d505aef8f 00007f6d505ae843 00007f6d505b1541 00007f6d505afe9e 00007f6d505b18cb
0812 08:02:32.767869 (+     5us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:33.804681 (+1036812us) raft_consensus.cc:1685] finished
0812 08:02:33.804684 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.804715 (+    31us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:32.767599 (+     0us) op_driver.cc:464] REPLICATION: finished
0812 08:02:32.771881 (+  4282us) write_op.cc:183] PREPARE: starting on tablet 21ed101d57de4eb983765b92682e2b5f
0812 08:02:32.772015 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:32.772022 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:32.772025 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:32.772110 (+    85us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:32.772130 (+    20us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:32.772135 (+     5us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:02:32.772195 (+    60us) tablet.cc:701] Row locks acquired
0812 08:02:32.772198 (+     3us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0812 08:02:32.767611 (+     0us) op_driver.cc:464] REPLICATION: finished
0812 08:02:32.773012 (+  5401us) write_op.cc:183] PREPARE: starting on tablet 21ed101d57de4eb983765b92682e2b5f
0812 08:02:32.773082 (+    70us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:32.773087 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:32.773088 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:32.773127 (+    39us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:32.773137 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:32.773142 (+     5us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:02:32.773169 (+    27us) tablet.cc:701] Row locks acquired
0812 08:02:32.773171 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"prepare.queue_time_us":5858,"prepare.run_cpu_time_us":3,"prepare.run_wall_time_us":3,"raft.queue_time_us":1215,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":2319104,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":6050,"child_traces":[["op",{"apply.queue_time_us":1031998,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":4791,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":631,"replication_time_us":615,"thread_start_us":251,"threads_started":2}],["op",{"apply.queue_time_us":1031849,"apply.run_cpu_time_us":296,"apply.run_wall_time_us":296,"num_ops":1,"partition_lock_wait_us":3,"prepare.queue_time_us":5731,"prepare.run_cpu_time_us":342,"prepare.run_wall_time_us":341,"replication_time_us":390,"thread_start_us":100,"threads_started":1}]]}
W20260812 08:02:33.812573 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37904: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
W20260812 08:02:33.874853 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.078s (SENT)
I20260812 08:02:33.874964 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:33.875140 24561 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260812 08:02:33.885594 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.078s (SENT)
I20260812 08:02:33.885700 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:33.886082 24612 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 08:02:33.907253 23570 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.099s (SENT)
I20260812 08:02:33.907361 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:33.907562 24612 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 08:02:33.907593 23570 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.099s (SENT)
I20260812 08:02:33.907713 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:33.907913 24612 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 08:02:33.975471 24561 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:33.975621 24561 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:33.976145 24561 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:33.976370 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.089s (SENT)
I20260812 08:02:33.976473 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:33.976656 24561 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 08:02:33.976656 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:33.976846 23670 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 8.
I20260812 08:02:33.977217 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:33.977418 24561 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 08:02:33.977471 24561 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:33.977537 24561 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:33.978457 24561 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:33.978860 24561 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:33.979280 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:33.979413 23670 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Advancing to term 10
I20260812 08:02:33.980261 23670 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 10.
I20260812 08:02:33.980609 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:33.980846 24561 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Leader election won for term 10
I20260812 08:02:33.980999 24561 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:33.981172 24561 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.19, Last appended by leader: 19, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:33.982617 24612 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:33.982728 24612 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:33.983134 24612 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:33.983095 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 8 to 10, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 10 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:33.983516 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:33.983675 23909 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 9 because replica has last-logged OpId of term: 8 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 13.
I20260812 08:02:34.003588 23670 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 8 index: 18. Preceding OpId from leader: term: 10 index: 20. (index mismatch)
I20260812 08:02:34.004165 24561 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260812 08:02:34.094889 24487 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:34.095048 24487 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:34.095495 24487 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:34.095918 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:34.096063 23670 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 8.
I20260812 08:02:34.096359 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:34.096552 24487 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 08:02:34.096617 24487 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:34.096670 24487 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:34.097815 24487 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:34.098196 24487 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:34.098560 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:34.098688 23670 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:34.099474 23670 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 9.
I20260812 08:02:34.099817 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:34.100003 24487 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader election won for term 9
I20260812 08:02:34.100104 24487 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:34.100245 24487 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 8.15, Last appended by leader: 15, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:34.101881 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 8 to 9, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 9 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:34.121171 23670 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 8 index: 13. Preceding OpId from leader: term: 9 index: 16. (index mismatch)
I20260812 08:02:34.121635 24487 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
W20260812 08:02:34.280375 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.148s (SENT)
I20260812 08:02:34.280480 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:34.280650 24612 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 08:02:34.285701 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.093s (SENT)
W20260812 08:02:34.297405 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.160s (SENT)
W20260812 08:02:34.311077 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.165s (SENT)
W20260812 08:02:34.337360 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:33.823250 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48482)
0812 08:02:33.823438 (+   188us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:33.823457 (+    19us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:33.823548 (+    91us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:34.337228 (+513680us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 08:02:34.337538 24011 meta_cache.cc:302] tablet 88f4b4af90a54b149dd1c350d0040d7c: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
W20260812 08:02:34.349205 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54130: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:34.945349 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:34.413862 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48486)
0812 08:02:34.414059 (+   197us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:34.414076 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:34.414183 (+   107us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:34.945229 (+531046us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 08:02:35.413905 23420 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.155.2:39949 (request call id 328) took 1032 ms (client timeout 99 ms). Trace:
W20260812 08:02:35.414031 23420 rpcz_store.cc:269] 0812 08:02:34.381349 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.413806 (+1032457us) service_pool.cc:211] Skipping call since client already timed out
0812 08:02:35.413878 (+    72us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260812 08:02:35.414270 23431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.155.2:39949 (request call id 329) took 1032 ms (client timeout 78 ms). Trace:
W20260812 08:02:35.414330 23431 rpcz_store.cc:269] 0812 08:02:34.381432 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.414238 (+1032806us) service_pool.cc:211] Skipping call since client already timed out
0812 08:02:35.414262 (+    24us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260812 08:02:35.414958 24504 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:35.415048 24504 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:35.415361 23437 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.415423 24504 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:35.415467 23429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.415668 23435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.415706 23440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.415763 23436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.415910 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.415948 23441 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 08:02:35.415992 23441 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:35.416064 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.416108 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.415668 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.416114 23441 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 8.15, Last appended by leader: 15, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:35.416706 23441 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:35.416915 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:35.417033 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.415719 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.417259 23438 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 08:02:35.417307 23438 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:35.417439 23438 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.19, Last appended by leader: 19, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:35.417559 23438 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Advancing to term 10
I20260812 08:02:35.417608 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.417990 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.418071 23441 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 8 index: 15. Preceding OpId from leader: term: 9 index: 16. (index mismatch)
I20260812 08:02:35.418174 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.418282 23434 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 9 because replica has last-logged OpId of term: 8 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 13.
W20260812 08:02:35.418459 24642 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:35.416852 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.22.155.1:46399)
0812 08:02:35.417518 (+   666us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:35.417779 (+   261us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:35.417800 (+    21us) server_negotiation.cc:408] Connection header received
0812 08:02:35.417849 (+    49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:35.417853 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:35.417932 (+    79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:35.418069 (+   137us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48466: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":87,"threads_started":1}
I20260812 08:02:35.418548 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.418597 23438 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 8 index: 19. Preceding OpId from leader: term: 10 index: 20. (index mismatch)
I20260812 08:02:35.418669 23430 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 9 because replica has last-logged OpId of term: 8 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 13.
I20260812 08:02:35.418895 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 6 index: 10 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:35.419013 24642 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:35.417041 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.22.155.1:46399)
0812 08:02:35.418682 (+  1641us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:35.418687 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:35.418703 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:35.418741 (+    38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:35.418744 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:35.418801 (+    57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:35.418923 (+   122us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48482: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":61,"threads_started":1}
I20260812 08:02:35.419044 23449 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 9 because replica has last-logged OpId of term: 8 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 10.
W20260812 08:02:35.419641 24642 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:35.417149 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.22.155.1:46399)
0812 08:02:35.419377 (+  2228us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:35.419381 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:35.419393 (+    12us) server_negotiation.cc:408] Connection header received
0812 08:02:35.419426 (+    33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:35.419429 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:35.419468 (+    39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:35.419562 (+    94us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48486: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2152,"thread_start_us":42,"threads_started":1}
W20260812 08:02:35.420758 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.188s (SENT)
I20260812 08:02:35.418915 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.424547 24646 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260812 08:02:35.425088 24646 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260812 08:02:35.426259 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:35.428126 23909 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 LEADER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 8. Current term is 10. Ops: []
W20260812 08:02:35.442468 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:35.443975 23342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:35.451138 24674 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:35.451242 24674 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:35.451462 23674 raft_consensus.cc:1217] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 8.13->[8.14-8.15]   Dedup: 8.15->[]
I20260812 08:02:35.451555 23674 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 8. Current term is 9. Ops: [8.14-8.15]
I20260812 08:02:35.452128 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:35.452523 24612 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:35.452629 24612 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:35.452733 24674 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:35.452975 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:35.453099 24612 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:35.453871 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:35.454056 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:35.457475 23658 raft_consensus.cc:1217] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 8.18->[8.19-8.19]   Dedup: 8.19->[]
I20260812 08:02:35.457563 23658 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 8. Current term is 10. Ops: [8.19-8.19]
W20260812 08:02:36.001389 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:35.483449 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37952)
0812 08:02:35.483686 (+   237us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:35.483703 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:35.483852 (+   149us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:36.001249 (+517397us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 08:02:36.001598 24010 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:36.484769 24649 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:36.484234 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.22.155.3:44619)
0812 08:02:36.484382 (+   148us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:36.484387 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:36.484404 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:36.484455 (+    51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:36.484462 (+     7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:36.484533 (+    71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:36.484659 (+   126us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37952: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 08:02:36.487922 24701 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:36.488353 24701 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:36.489254 24701 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:36.489295 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
W20260812 08:02:36.513209 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:36.003293 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:48492)
0812 08:02:36.003488 (+   195us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:36.003503 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:36.003593 (+    90us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:36.513119 (+509526us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 08:02:36.519317 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54146: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:37.089371 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:36.584232 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:37996)
0812 08:02:36.584401 (+   169us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:36.584415 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:36.584487 (+    72us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:37.089254 (+504767us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 08:02:37.089602 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:37.587077 23570 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.092s (SENT)
I20260812 08:02:37.587177 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
W20260812 08:02:37.587458 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.097s (SENT)
I20260812 08:02:37.587515 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:37.588009 24728 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 08:02:37.588253 24728 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260812 08:02:36.552027 23336 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.105s (SENT) [suppressed 6 similar messages]
W20260812 08:02:37.588562 23336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.155.2:33271, user_credentials={real_user=slave}} blocked reactor thread for 1.03647e+06us
I20260812 08:02:37.586339 24555 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.589563 24555 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:37.589926 24555 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:37.590337 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:37.590677 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:37.590821 24730 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260812 08:02:37.592288 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:37.592427 23650 raft_consensus.cc:2410] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 10 because replica has last-logged OpId of term: 9 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 18.
I20260812 08:02:37.592873 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:37.593079 23685 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for term 7 because replica has last-logged OpId of term: 6 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 13.
I20260812 08:02:37.593384 23336 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:37.593629 24555 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260812 08:02:37.594307 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:37.092307 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:54156)
0812 08:02:37.092460 (+   153us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:37.092475 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:37.092559 (+    84us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:37.594194 (+501635us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 25 requested bytes
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 08:02:37.586360 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:37.586499 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:37.586786 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:37.594696 24726 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:37.585015 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.22.155.2:33271)
0812 08:02:37.593899 (+  8884us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:37.593903 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:37.593919 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:37.593963 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:37.593967 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:37.594015 (+    48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:37.594563 (+   548us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54156: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":8804,"thread_start_us":71,"threads_started":1}
I20260812 08:02:37.587221 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 13 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:37.585999 23341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:37.596720 24734 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:37.595299 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.22.155.1:46399)
0812 08:02:37.596384 (+  1085us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:37.596388 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:37.596402 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:37.596443 (+    41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:37.596446 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:37.596496 (+    50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:37.596615 (+   119us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48492: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":57,"threads_started":1}
I20260812 08:02:37.662873 24555 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.663030 24555 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.663480 24555 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:37.664014 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:37.664152 23685 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 10.
I20260812 08:02:37.664489 23336 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.664703 24555 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 08:02:37.664784 24555 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.664848 24555 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:37.665834 24555 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.666209 24555 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:37.666632 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:37.666743 23685 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:37.667603 23685 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 11.
I20260812 08:02:37.667953 23336 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.668144 24555 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Leader election won for term 11
I20260812 08:02:37.668241 24555 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:37.668399 24555 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.670244 23212 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 10 to 11, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 11 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:37.679111 24730 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.679234 24730 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.679694 24730 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:37.680105 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:37.680227 23441 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 6.
I20260812 08:02:37.680634 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.680833 24730 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 08:02:37.680917 24730 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.681006 24730 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:02:37.682260 24728 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.682359 24728 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:37.682359 24730 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.682757 24730 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:37.682754 24728 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:37.683220 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:37.683224 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:37.683380 23441 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:02:37.683388 23428 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 9.
I20260812 08:02:37.683686 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.683890 24728 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 08:02:37.683949 24728 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:37.683979 24728 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:37.684518 23441 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 7.
I20260812 08:02:37.684837 24728 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:37.684892 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.685135 24730 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Leader election won for term 7
I20260812 08:02:37.685343 24730 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:37.685376 24728 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:37.685510 24730 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:37.685778 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:37.685899 23441 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:37.686853 23441 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 10.
I20260812 08:02:37.687163 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:37.687381 24730 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader election won for term 10
I20260812 08:02:37.687355 23212 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 6 to 7, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 7 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:37.687503 24730 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:37.687629 24730 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 9.19, Last appended by leader: 19, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:37.690760 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 9 to 10, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 10 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:37.697718 23685 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 11 index: 24. (index mismatch)
I20260812 08:02:37.698311 24637 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260812 08:02:37.709236 23441 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 6 index: 14. Preceding OpId from leader: term: 7 index: 17. (index mismatch)
I20260812 08:02:37.709649 24730 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260812 08:02:37.719097 23441 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 9 index: 18. Preceding OpId from leader: term: 10 index: 20. (index mismatch)
I20260812 08:02:37.719537 24743 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
W20260812 08:02:37.790370 23341 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.097s (SENT)
I20260812 08:02:37.790498 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:37.790709 24555 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260812 08:02:37.841883 23341 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.087s (SENT)
W20260812 08:02:37.853822 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.086s (SENT)
W20260812 08:02:37.873634 23341 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.104s (SENT)
W20260812 08:02:37.928153 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.122s (SENT)
W20260812 08:02:37.976078 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.146s (SENT)
W20260812 08:02:37.977487 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.145s (SENT)
W20260812 08:02:38.117249 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:37.602123 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:42288)
0812 08:02:37.602276 (+   153us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:37.602289 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:37.602350 (+    61us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:38.117147 (+514797us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 08:02:38.117448 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:38.125849 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36658: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:38.657262 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:38.150268 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:42296)
0812 08:02:38.150433 (+   165us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:38.150448 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:38.150515 (+    67us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:38.657163 (+506648us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 08:02:38.665637 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36666: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
I20260812 08:02:38.678035 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.678098 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.678423 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.678668 23911 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 LEADER]: Stepping down as leader of term 10
I20260812 08:02:38.678704 23890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 18 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.679032 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.679255 23887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.679276 23907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.679390 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.679071 23889 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.679404 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.679710 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.680852 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:38.678726 23911 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:38.681073 23911 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:38.681226 23911 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:38.682000 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 7 candidate_status { last_received { term: 6 index: 16 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:38.682421 23911 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 11 index: 24. (index mismatch)
I20260812 08:02:38.683219 23913 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Stepping down as leader of term 6
I20260812 08:02:38.683293 23913 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:38.683424 23913 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 6.16, Last appended by leader: 16, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:38.683521 23913 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:02:38.683982 23685 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 LEADER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 9. Current term is 10. Ops: []
I20260812 08:02:38.684031 23650 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 6. Current term is 7. Ops: []
I20260812 08:02:38.684546 24487 consensus_queue.cc:1059] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 9.19, Next index: 20, Last known committed idx: 22, Time since last communication: 0.000s
W20260812 08:02:38.684644 24785 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:38.682612 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.22.155.3:44619)
0812 08:02:38.684297 (+  1685us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:38.684302 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:38.684316 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:38.684358 (+    42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:38.684361 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:38.684407 (+    46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:38.684528 (+   121us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42296: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1600,"thread_start_us":58,"threads_started":1}
I20260812 08:02:38.684681 24487 consensus_queue.cc:1243] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Peer 9c1d86a754e645269bc0796059331944 log is divergent from this leader: its last log entry 10.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260812 08:02:38.684741 23918 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 LEADER]: Stepping down as leader of term 9
I20260812 08:02:38.684790 23918 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:38.684909 23918 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 9.19, Last appended by leader: 19, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:38.685166 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 10 candidate_status { last_received { term: 9 index: 19 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
W20260812 08:02:38.685182 24785 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:38.683295 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.22.155.3:44619)
0812 08:02:38.684820 (+  1525us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:38.684824 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:38.684835 (+    11us) server_negotiation.cc:408] Connection header received
0812 08:02:38.684869 (+    34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:38.684881 (+    12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:38.684925 (+    44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:38.685085 (+   160us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37996: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1473,"thread_start_us":50,"threads_started":1}
I20260812 08:02:38.685093 23918 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Advancing to term 10
W20260812 08:02:38.685724 24785 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:38.683395 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.22.155.3:44619)
0812 08:02:38.685403 (+  2008us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:38.685407 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:38.685419 (+    12us) server_negotiation.cc:408] Connection header received
0812 08:02:38.685513 (+    94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:38.685516 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:38.685556 (+    40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:38.685646 (+    90us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42288: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":45,"threads_started":1}
I20260812 08:02:38.684521 23913 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 6 index: 16. Preceding OpId from leader: term: 7 index: 17. (index mismatch)
I20260812 08:02:38.686334 23918 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 9 index: 19. Preceding OpId from leader: term: 10 index: 20. (index mismatch)
I20260812 08:02:38.686334 24728 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260812 08:02:38.686532 24487 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260812 08:02:38.686930 24728 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260812 08:02:39.233373 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:38.710061 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:39008)
0812 08:02:38.710258 (+   197us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:38.710281 (+    23us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:38.710363 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:39.233249 (+522886us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 08:02:39.233681 24009 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:39.712705 23438 raft_consensus.cc:1217] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 9.18->[9.19-9.19]   Dedup: 9.19->[]
I20260812 08:02:39.712849 23438 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 9. Current term is 10. Ops: [9.19-9.19]
W20260812 08:02:39.713088 24799 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:39.711142 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.22.155.2:33271)
0812 08:02:39.712567 (+  1425us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:39.712571 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:39.712587 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:39.712639 (+    52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:39.712644 (+     5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:39.712712 (+    68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:39.712891 (+   179us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39008: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":50,"threads_started":1}
I20260812 08:02:39.713912 24555 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:39.714017 24555 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:39.714426 24555 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:39.714736 23449 raft_consensus.cc:1217] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 6.14->[6.15-6.16]   Dedup: 6.16->[]
I20260812 08:02:39.714823 23449 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 6. Current term is 7. Ops: [6.15-6.16]
I20260812 08:02:39.717257 24802 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260812 08:02:39.717417 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:39.718267 24730 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:39.718361 24730 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:39.718703 24730 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:39.719017 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:39.723834 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36682: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:39.757440 23801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260812 08:02:39.757580 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:39.757907 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:39.758060 23911 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 10.
I20260812 08:02:39.758060 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:39.758596 24487 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:39.758713 24487 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:39.759007 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 9c1d86a754e645269bc0796059331944
I20260812 08:02:39.759150 24487 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:39.759222 24487 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:39.759261 24802 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 08:02:39.759318 24802 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:39.759282 24487 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:39.759359 24802 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:39.759554 24777 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:39.759614 24487 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:39.759632 24777 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:39.760694 24802 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:39.760814 24777 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:39.761070 24802 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:39.762328 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:39.762454 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:39.762575 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:39.765128 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:39.765308 24795 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260812 08:02:40.321326 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:39.790408 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:39040)
0812 08:02:39.790609 (+   201us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:39.790625 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:39.790713 (+    88us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:40.321214 (+530501us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":81}
W20260812 08:02:40.321556 24009 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:40.790867 24728 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:40.791020 24728 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:40.791560 24728 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:40.793656 23428 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 10. Current term is 11. Ops: []
W20260812 08:02:40.793735 24841 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:40.791905 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.22.155.2:33271)
0812 08:02:40.792420 (+   515us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:40.792425 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:40.792442 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:40.792491 (+    49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:40.792495 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:40.792548 (+    53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:40.792674 (+   126us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39040: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":79,"threads_started":1}
I20260812 08:02:40.793776 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:40.793926 23434 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 11: Already voted for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in this term.
I20260812 08:02:40.794222 24730 consensus_queue.cc:1059] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: INVALID_TERM, Last received: 10.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260812 08:02:40.794349 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:40.794370 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:40.794425 24730 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 LEADER]: Stepping down as leader of term 10
I20260812 08:02:40.794466 24730 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:40.794605 24730 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:40.794767 24730 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:40.795320 24555 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:40.795403 24555 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:40.795701 24555 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:40.796073 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:40.796267 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:40.797302 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:40.801769 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36692: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:40.816712 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39048: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
I20260812 08:02:40.878582 24795 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:40.878706 24795 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:40.879184 24795 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:40.879551 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:40.879673 23438 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 11.
I20260812 08:02:40.879936 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:40.880128 24795 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260812 08:02:40.880196 24795 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:40.880225 24795 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:02:40.881162 24795 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:40.881552 24795 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:40.881999 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:40.882105 23438 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:02:40.882887 23438 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 12.
I20260812 08:02:40.883194 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:40.883385 24795 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Leader election won for term 12
I20260812 08:02:40.883473 24795 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:40.883623 24795 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:40.885278 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } }
W20260812 08:02:40.887900 23568 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.096s (SENT) [suppressed 10 similar messages]
I20260812 08:02:40.910794 23438 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 12 index: 24. (index mismatch)
I20260812 08:02:40.911360 24795 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260812 08:02:40.959787 23341 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.082s (SENT)
I20260812 08:02:40.959923 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:40.960173 24842 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260812 08:02:40.985349 23568 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.096s (SENT)
I20260812 08:02:40.985457 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:40.985625 24795 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260812 08:02:41.016083 23341 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.109s (SENT)
I20260812 08:02:41.016189 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:41.016382 24842 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 08:02:41.068873 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.093s (SENT)
W20260812 08:02:41.096453 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.108s (SENT)
W20260812 08:02:41.349243 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:40.824735 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:42304)
0812 08:02:40.824912 (+   177us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:40.824926 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:40.825055 (+   129us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:41.349149 (+524094us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 08:02:41.349416 24010 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:41.362177 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36708: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:41.399947 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.092s (SENT) [suppressed 10 similar messages]
I20260812 08:02:41.400971 23907 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Advancing to term 12
I20260812 08:02:41.401086 23908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:41.401166 23911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:41.401257 23911 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 7.
I20260812 08:02:41.401438 23913 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:41.401482 23889 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:41.401710 23887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:41.401921 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:41.402029 23903 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 11.
I20260812 08:02:41.402238 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:41.402323 23907 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 12 index: 24. (index mismatch)
I20260812 08:02:41.402467 23890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:41.402560 23890 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 12.
I20260812 08:02:41.402746 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:41.403107 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
W20260812 08:02:41.403228 24888 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:41.402635 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.22.155.3:44619)
0812 08:02:41.402848 (+   213us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:41.402853 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:41.402868 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:41.402913 (+    45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:41.402917 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:41.402980 (+    63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:41.403122 (+   142us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42304: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260812 08:02:41.405375 24730 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260812 08:02:41.953320 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:41.432184 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:39080)
0812 08:02:41.432406 (+   222us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:41.432423 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:41.432516 (+    93us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:41.953205 (+520689us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 08:02:42.432770 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:42.432924 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 8 candidate_status { last_received { term: 7 index: 18 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:42.433032 23449 raft_consensus.cc:2368] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for earlier term 11. Current term is 12.
I20260812 08:02:42.432770 24842 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:42.433208 24842 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:42.433324 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 24 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:42.433538 24900 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:42.433624 24900 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:42.433712 24842 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:42.435060 23430 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 8 because replica has last-logged OpId of term: 7 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 18.
I20260812 08:02:42.433948 24900 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
W20260812 08:02:42.436585 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.092s (SENT)
I20260812 08:02:42.436657 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
W20260812 08:02:42.436892 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.097s (SENT)
I20260812 08:02:42.436934 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:42.437079 24901 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260812 08:02:42.437129 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.103s (SENT)
I20260812 08:02:42.437166 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:42.437189 24901 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260812 08:02:42.437273 24901 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:02:42.438329 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:42.438442 23918 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 7.
I20260812 08:02:42.438633 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:42.438704 23906 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 12.
I20260812 08:02:42.440248 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:42.440400 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:42.440558 24884 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 08:02:42.440598 24884 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:42.440644 24884 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:42.441736 24884 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:42.442071 24884 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:42.442179 24898 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260812 08:02:42.442242 24898 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:42.442284 24898 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Advancing to term 13
W20260812 08:02:42.442615 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42308: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:42.443212 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:42.443305 23906 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:42.443295 24898 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:42.443626 24898 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:42.444253 23906 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 8.
I20260812 08:02:42.445003 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:42.445195 24898 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 08:02:42.445256 24898 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:42.445365 24898 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 7.21, Last appended by leader: 21, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:42.446370 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:42.446477 23909 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:02:42.446727 23212 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 7 to 8, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 8 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:42.447444 23909 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 13.
I20260812 08:02:42.447749 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:42.447949 24884 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Leader election won for term 13
I20260812 08:02:42.448028 24884 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:42.448138 24884 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 12.25, Last appended by leader: 25, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:42.451753 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 12 to 13, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 13 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:42.457050 23909 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 12 index: 25. Preceding OpId from leader: term: 13 index: 27. (index mismatch)
I20260812 08:02:42.457495 24884 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260812 08:02:42.475399 23909 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 7 index: 21. Preceding OpId from leader: term: 8 index: 22. (index mismatch)
I20260812 08:02:42.475900 24898 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260812 08:02:42.515578 23336 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.080s (SENT) [suppressed 5 similar messages]
W20260812 08:02:42.596429 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.080s (SENT)
W20260812 08:02:42.622849 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.093s (SENT)
W20260812 08:02:42.626194 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.090s (SENT)
W20260812 08:02:42.637988 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.097s (SENT)
W20260812 08:02:42.977281 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:42.463001 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:39082)
0812 08:02:42.463163 (+   162us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:42.463182 (+    19us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:42.463261 (+    79us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:42.977163 (+513902us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 08:02:42.977490 24009 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:42.995761 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:42.995995 24730 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:42.996105 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:42.996381 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:42.996635 23680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:42.997080 23649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:42.997169 23648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 8 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:42.997400 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:42.997658 23655 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Stepping down as leader of term 12
I20260812 08:02:42.997736 23655 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:42.998020 23655 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 12.25, Last appended by leader: 25, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:42.998194 23655 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:02:42.998560 23906 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 7. Current term is 8. Ops: []
I20260812 08:02:42.998919 23909 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 12. Current term is 13. Ops: []
I20260812 08:02:42.999069 24945 consensus_queue.cc:1059] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: INVALID_TERM, Last received: 7.21, Next index: 22, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 08:02:42.999477 23655 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 12 index: 25. Preceding OpId from leader: term: 13 index: 27. (index mismatch)
I20260812 08:02:42.996081 24730 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:42.999665 24795 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Stepping down as leader of term 7
I20260812 08:02:42.999732 24795 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:42.999861 24795 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 7.21, Last appended by leader: 21, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:42.999959 24795 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:02:43.000031 24730 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
W20260812 08:02:43.000187 24949 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:42.997408 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.22.155.2:33271)
0812 08:02:42.999373 (+  1965us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:42.999379 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:42.999398 (+    19us) server_negotiation.cc:408] Connection header received
0812 08:02:42.999464 (+    66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:42.999467 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:42.999526 (+    59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:42.999665 (+   139us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1729,"thread_start_us":60,"threads_started":1}
W20260812 08:02:43.000598 24949 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:42.997524 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.22.155.2:33271)
0812 08:02:43.000343 (+  2819us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:43.000347 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:43.000360 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:43.000402 (+    42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:43.000405 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:43.000439 (+    34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:43.000533 (+    94us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39082: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2759,"thread_start_us":40,"threads_started":1}
I20260812 08:02:43.001528 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:43.001613 23668 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 7 index: 21. Preceding OpId from leader: term: 8 index: 22. (index mismatch)
I20260812 08:02:43.001879 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:43.002000 23685 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 13.
W20260812 08:02:43.553345 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:43.027492 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:36740)
0812 08:02:43.027653 (+   161us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:43.027670 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:43.027762 (+    92us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:43.553232 (+525470us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 08:02:44.028226 23438 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 7. Current term is 8. Ops: []
I20260812 08:02:44.028170 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:44.028855 24952 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260812 08:02:44.028899 24957 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260812 08:02:44.030227 24965 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:44.029581 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.22.155.1:46399)
0812 08:02:44.029813 (+   232us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:44.029817 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:44.029834 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:44.029886 (+    52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:44.029892 (+     6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:44.029957 (+    65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:44.030086 (+   129us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36740: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
I20260812 08:02:44.033207 24966 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:44.033349 24966 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:44.033751 24966 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:44.033886 24966 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:44.033987 24966 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:44.034142 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:44.034375 24966 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:44.034492 24966 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:44.034588 24966 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:44.034879 24966 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:44.034864 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 24 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:44.035527 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:44.039109 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42322: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:44.073056 24945 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:44.073179 24945 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:44.073560 24945 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
W20260812 08:02:44.073755 23570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:44.073899 24945 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:44.073974 24945 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:44.074409 24945 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:44.074797 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:44.077040 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:44.077423 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:44.078435 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:44.078684 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 24 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:44.080308 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:44.080577 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:44.080775 24969 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:02:44.080793 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:44.080972 24969 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260812 08:02:44.081069 24969 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 08:02:44.609401 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:44.105292 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:36774)
0812 08:02:44.105449 (+   157us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:44.105464 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:44.105551 (+    87us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:44.609257 (+503706us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 08:02:44.609640 24011 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
W20260812 08:02:45.105387 23570 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.088s (SENT)
I20260812 08:02:45.105536 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:45.106189 24945 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:02:45.108582 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.109122 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.109397 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.108898 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.111191 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:45.111428 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
W20260812 08:02:45.113135 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:44.612330 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:39092)
0812 08:02:44.612542 (+   212us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:44.612556 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:44.612634 (+    78us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:45.109872 (+497238us) client_negotiation.cc:281] Received NEGOTIATE NegotiatePB response
0812 08:02:45.109877 (+     5us) client_negotiation.cc:404] Received NEGOTIATE response from server
0812 08:02:45.109888 (+    11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0812 08:02:45.110165 (+   277us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0812 08:02:45.110176 (+    11us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0812 08:02:45.111286 (+  1110us) client_negotiation.cc:281] Received TLS_HANDSHAKE NegotiatePB response
0812 08:02:45.111290 (+     4us) client_negotiation.cc:543] Received TLS_HANDSHAKE response from server
0812 08:02:45.111897 (+   607us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0812 08:02:45.111908 (+    11us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0812 08:02:45.111991 (+    83us) client_negotiation.cc:564] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 08:02:45.112025 (+    34us) client_negotiation.cc:262] Sending TOKEN_EXCHANGE NegotiatePB request
0812 08:02:45.112975 (+   950us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 15 requested bytes
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 08:02:45.113111 24945 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260812 08:02:45.113247 24945 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 08:02:45.113462 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 7 index: 21 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:45.113794 25003 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:45.106941 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.22.155.2:33271)
0812 08:02:45.107535 (+   594us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:45.107540 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:45.109609 (+  2069us) server_negotiation.cc:408] Connection header received
0812 08:02:45.109659 (+    50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:45.109662 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:45.109714 (+    52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:45.110280 (+   566us) server_negotiation.cc:227] Negotiated authn=TOKEN
0812 08:02:45.110384 (+   104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 08:02:45.111145 (+   761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 08:02:45.112437 (+  1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 08:02:45.112634 (+   197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 08:02:45.112672 (+    38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0812 08:02:45.112770 (+    98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0812 08:02:45.113666 (+   896us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39092: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":57,"threads_started":1}
W20260812 08:02:45.113900 25005 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:45.109957 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.22.155.1:46399)
0812 08:02:45.113392 (+  3435us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:45.113396 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:45.113409 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:45.113452 (+    43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:45.113455 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:45.113504 (+    49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:45.113621 (+   117us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36774: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":3355,"thread_start_us":79,"threads_started":1}
W20260812 08:02:45.633364 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:45.120027 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:42326)
0812 08:02:45.120201 (+   174us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:45.120218 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:45.120314 (+    96us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:45.633238 (+512924us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 08:02:45.633563 24010 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
I20260812 08:02:45.669957 24966 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:45.670094 24966 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:45.670266 24967 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:45.670353 24967 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:45.670548 24966 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:45.670666 24967 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:45.671128 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:45.671176 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:45.671904 24967 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:45.671911 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.671988 24967 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:45.672307 24967 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:45.672637 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:45.672683 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:45.672698 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:45.673094 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:45.673714 24966 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260812 08:02:45.673856 24966 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 08:02:45.681092 25022 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:45.670276 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.22.155.3:44619)
0812 08:02:45.680686 (+ 10410us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:45.680692 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:45.680708 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:45.680756 (+    48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:45.680760 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:45.680811 (+    51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:45.680964 (+   153us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42326: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":10322,"thread_start_us":82,"threads_started":1}
W20260812 08:02:46.209365 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:45.702107 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:36806)
0812 08:02:45.702283 (+   176us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:45.702299 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:45.702402 (+   103us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:46.209252 (+506850us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 08:02:46.704510 23449 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.155.3:39087 (request call id 110) took 1032 ms (client timeout 109 ms). Trace:
W20260812 08:02:46.704602 23449 rpcz_store.cc:269] 0812 08:02:45.671831 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:46.704459 (+1032628us) service_pool.cc:211] Skipping call since client already timed out
0812 08:02:46.704494 (+    35us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260812 08:02:46.705838 25038 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:46.705946 25038 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:46.706236 25037 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:46.706328 25037 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:46.706532 25038 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:46.706606 25038 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:46.706664 25037 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:46.706784 25038 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:46.707233 25038 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:46.708987 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:46.709270 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:46.709337 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:46.711946 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39102: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.712522 25041 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:46.709831 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.22.155.1:46399)
0812 08:02:46.712091 (+  2260us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:46.712095 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:46.712108 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:46.712150 (+    42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:46.712154 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:46.712216 (+    62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:46.712363 (+   147us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36806: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2166,"thread_start_us":71,"threads_started":1}
W20260812 08:02:46.783790 23568 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.076s (SENT) [suppressed 5 similar messages]
W20260812 08:02:46.860975 23568 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.076s (SENT)
I20260812 08:02:46.861092 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:46.861361 25026 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260812 08:02:46.869724 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.081s (SENT)
I20260812 08:02:46.869817 23568 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:46.869971 25026 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260812 08:02:46.909703 23568 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.100s (SENT)
I20260812 08:02:46.909813 23568 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:46.910029 25026 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 08:02:47.233368 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:46.715961 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:42328)
0812 08:02:46.716112 (+   151us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:46.716134 (+    22us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:46.716199 (+    65us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:47.233211 (+517012us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 08:02:47.233578 24010 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:47.269507 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.109s (SENT) [suppressed 2 similar messages]
I20260812 08:02:47.270529 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 25 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:47.270687 23887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.270802 23887 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 8.
I20260812 08:02:47.270982 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:47.271054 23890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.271307 25065 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:47.271320 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.271397 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.271440 25065 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:47.271674 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.271989 23891 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 13.
I20260812 08:02:47.272324 25065 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:47.272562 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:47.273128 25065 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:47.273208 25065 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:47.273738 25065 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:47.273824 25065 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:02:47.274004 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:47.278127 23907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:47.278759 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:47.279091 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
W20260812 08:02:47.280507 25070 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:47.278551 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.22.155.3:44619)
0812 08:02:47.280116 (+  1565us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:47.280122 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:47.280139 (+    17us) server_negotiation.cc:408] Connection header received
0812 08:02:47.280185 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:47.280188 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:47.280232 (+    44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:47.280367 (+   135us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42328: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":53,"threads_started":1}
W20260812 08:02:47.809377 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:47.301187 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:36844)
0812 08:02:47.301343 (+   156us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:47.301358 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:47.301445 (+    87us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:47.809267 (+507822us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 08:02:48.301487 25065 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.302343 25065 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.303921 25065 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:48.305424 25078 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.305572 25078 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.306097 25078 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:48.306227 25078 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.306351 25078 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.306511 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:48.306663 25085 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.306964 25085 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:48.306692 23655 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 12 because replica has last-logged OpId of term: 11 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 26.
I20260812 08:02:48.307041 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 29 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:48.307399 23658 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 9 because replica has last-logged OpId of term: 8 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 29.
I20260812 08:02:48.306692 25078 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:48.307564 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:48.307667 23656 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 13.
I20260812 08:02:48.307732 25085 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:48.307902 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:48.308100 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9
I20260812 08:02:48.308262 25065 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 08:02:48.308310 23907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:48.308363 25065 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260812 08:02:48.308395 25065 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.308430 25065 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:02:48.308444 23907 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 8.
I20260812 08:02:48.308650 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:48.308992 23568 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:48.309140 25085 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 08:02:48.309178 25085 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.309213 25085 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:48.309561 25065 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:48.309902 25065 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:48.310035 23918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 27 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:48.310133 23918 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 11.
I20260812 08:02:48.310243 25085 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.310616 23906 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 14: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:48.310729 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:48.310824 23656 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:02:48.310925 25085 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:48.311755 23656 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14.
I20260812 08:02:48.312078 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:48.312078 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:48.312232 24966 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Leader election won for term 14
I20260812 08:02:48.312330 24966 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:48.312456 24966 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 13.27, Last appended by leader: 27, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:48.312835 25085 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260812 08:02:48.312888 25085 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:48.312927 25085 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:02:48.313798 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:48.313894 23906 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:48.313849 25085 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.314196 25085 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:48.314399 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 13 to 14, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 14 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:48.314841 23906 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 9.
W20260812 08:02:48.315369 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:47.813429 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:52974)
0812 08:02:47.813596 (+   167us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:47.813612 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:47.813698 (+    86us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:48.307075 (+493377us) client_negotiation.cc:281] Received NEGOTIATE NegotiatePB response
0812 08:02:48.307079 (+     4us) client_negotiation.cc:404] Received NEGOTIATE response from server
0812 08:02:48.307092 (+    13us) client_negotiation.cc:190] Negotiated authn=TOKEN
0812 08:02:48.307380 (+   288us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0812 08:02:48.307390 (+    10us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0812 08:02:48.310994 (+  3604us) client_negotiation.cc:281] Received TLS_HANDSHAKE NegotiatePB response
0812 08:02:48.311000 (+     6us) client_negotiation.cc:543] Received TLS_HANDSHAKE response from server
0812 08:02:48.311639 (+   639us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0812 08:02:48.311651 (+    12us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0812 08:02:48.311762 (+   111us) client_negotiation.cc:564] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 08:02:48.311796 (+    34us) client_negotiation.cc:262] Sending TOKEN_EXCHANGE NegotiatePB request
0812 08:02:48.315229 (+  3433us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 15 requested bytes
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 08:02:48.315452 25081 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:48.304528 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.22.155.2:33271)
0812 08:02:48.304788 (+   260us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:48.304793 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:48.304809 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:48.304858 (+    49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:48.304861 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:48.304911 (+    50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:48.305029 (+   118us) server_negotiation.cc:227] Negotiated authn=TOKEN
0812 08:02:48.307935 (+  2906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 08:02:48.308679 (+   744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0812 08:02:48.312320 (+  3641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0812 08:02:48.312552 (+   232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0812 08:02:48.312595 (+    43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0812 08:02:48.312673 (+    78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0812 08:02:48.312767 (+    94us) server_negotiation.cc:1036] Waiting for connection context
0812 08:02:48.315302 (+  2535us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52974: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
I20260812 08:02:48.315474 23906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 27 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
W20260812 08:02:48.315496 24009 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 15 requested bytes
I20260812 08:02:48.315567 23906 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:02:48.315676 23568 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:48.315899 25082 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 08:02:48.315991 25082 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:48.316118 25082 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 8.31, Last appended by leader: 31, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.317761 23210 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 8 to 9, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 9 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:48.317934 23906 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 12.
I20260812 08:02:48.319214 23568 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:48.319444 25082 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Leader election won for term 12
I20260812 08:02:48.319533 25082 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:48.319666 25082 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 11.27, Last appended by leader: 27, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:48.322098 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 11 to 12, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 12 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:48.324826 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51446: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:48.337590 23656 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 13 index: 27. Preceding OpId from leader: term: 14 index: 28. (index mismatch)
I20260812 08:02:48.338104 25065 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260812 08:02:48.339381 23906 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 8 index: 29. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
I20260812 08:02:48.339850 25082 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 29, Time since last communication: 0.000s
I20260812 08:02:48.351001 23906 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 11 index: 26. Preceding OpId from leader: term: 12 index: 28. (index mismatch)
I20260812 08:02:48.351604 25078 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
W20260812 08:02:48.929399 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:48.397201 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:53006)
0812 08:02:48.397376 (+   175us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:48.397391 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:48.397475 (+    84us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:48.929281 (+531806us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 08:02:49.399560 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.399684 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.399988 23450 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 LEADER]: Stepping down as leader of term 11
I20260812 08:02:49.400090 23450 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:49.400235 23450 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 11.27, Last appended by leader: 27, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:49.400372 23450 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Advancing to term 12
W20260812 08:02:49.400514 23449 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.155.2:39949 (request call id 513) took 1035 ms (client timeout 97 ms). Trace:
W20260812 08:02:49.400569 23449 rpcz_store.cc:269] 0812 08:02:49.398816 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:49.400477 (+  1661us) service_pool.cc:211] Skipping call since client already timed out
0812 08:02:49.400504 (+    27us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260812 08:02:49.400933 23420 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Stepping down as leader of term 13
I20260812 08:02:49.401043 23420 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:49.401194 23420 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 13.27, Last appended by leader: 27, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:49.401366 23420 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:02:49.401505 23450 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 11 index: 27. Preceding OpId from leader: term: 12 index: 28. (index mismatch)
I20260812 08:02:49.401717 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.402453 23420 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 13 index: 27. Preceding OpId from leader: term: 14 index: 28. (index mismatch)
I20260812 08:02:49.402583 23438 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 12.
I20260812 08:02:49.399557 23428 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 08:02:49.402757 23428 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:49.402877 23428 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 8.31, Last appended by leader: 31, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:49.403059 23428 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:02:49.404067 23428 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 8 index: 31. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
W20260812 08:02:49.404186 25127 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:49.403335 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.22.155.2:33271)
0812 08:02:49.403757 (+   422us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:49.403762 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:49.403777 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:49.403829 (+    52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:49.403835 (+     6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:49.403914 (+    79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:49.404061 (+   147us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53006: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":60,"threads_started":1}
I20260812 08:02:49.402760 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.404381 25082 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260812 08:02:49.404887 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.405040 23434 raft_consensus.cc:2393] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 12: Already voted for candidate 9c1d86a754e645269bc0796059331944 in this term.
I20260812 08:02:49.405186 23430 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 9.
I20260812 08:02:49.405237 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.405272 25082 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20260812 08:02:49.405953 25123 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:49.400905 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.22.155.1:46399)
0812 08:02:49.405596 (+  4691us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:49.405600 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:49.405615 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:49.405655 (+    40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:49.405659 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:49.405705 (+    46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:49.405820 (+   115us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36844: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":4617,"thread_start_us":76,"threads_started":1}
I20260812 08:02:49.406032 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.406132 23430 raft_consensus.cc:2376] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 9. Re-sending same reply.
I20260812 08:02:49.406250 23441 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14.
I20260812 08:02:49.407459 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.407891 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 12 candidate_status { last_received { term: 11 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.408155 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.408258 23438 raft_consensus.cc:2376] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 9. Re-sending same reply.
I20260812 08:02:49.408622 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.408759 23450 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 14: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:49.409237 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:49.409524 25082 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260812 08:02:49.414108 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49174: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
I20260812 08:02:49.504678 25139 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:49.504802 25139 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:49.505245 25139 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:49.505599 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:49.505720 23450 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 14.
I20260812 08:02:49.505993 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:49.506186 25139 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260812 08:02:49.506249 25139 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 8811a4670c574f729b04cebef39bc7aa)
I20260812 08:02:49.506278 25139 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:02:49.507126 25139 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:49.507479 25139 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
I20260812 08:02:49.507898 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:49.508028 23450 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:02:49.508809 23450 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1d86a754e645269bc0796059331944 in term 15.
I20260812 08:02:49.509140 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:49.509351 25139 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 FOLLOWER]: Leader election won for term 15
I20260812 08:02:49.509461 25139 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 LEADER]: Becoming Leader. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:49.509600 25139 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 14.28, Last appended by leader: 28, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:49.511209 23212 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 reported cstate change: term changed from 14 to 15, leader changed from 8811a4670c574f729b04cebef39bc7aa (127.22.155.3) to 9c1d86a754e645269bc0796059331944 (127.22.155.2). New cstate: current_term: 15 leader_uuid: "9c1d86a754e645269bc0796059331944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:49.530143 23450 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 13 index: 27. Preceding OpId from leader: term: 15 index: 29. (index mismatch)
I20260812 08:02:49.530615 25082 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
W20260812 08:02:49.727186 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.110s (SENT)
W20260812 08:02:49.727371 23568 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619): Timed out: RequestConsensusVote RPC to 127.22.155.3:44619 timed out after 0.109s (SENT)
W20260812 08:02:49.921353 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:49.417216 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:51456)
0812 08:02:49.417397 (+   181us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:49.417411 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:49.417478 (+    67us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:49.921219 (+503741us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 08:02:49.921525 24010 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
W20260812 08:02:49.940887 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49186: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
W20260812 08:02:50.465270 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:49.946863 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.3:44619 (local address 127.0.0.1:51472)
0812 08:02:49.947030 (+   167us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:49.947045 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:49.947136 (+    91us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:50.465170 (+518034us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.3:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 08:02:50.489329 23909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:50.489945 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:50.490235 23907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:50.493942 25065 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:50.494045 25065 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:50.494288 23656 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 LEADER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 14. Current term is 15. Ops: []
I20260812 08:02:50.494450 25065 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:50.494534 25065 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260812 08:02:50.494853 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 9 index: 33 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:50.495349 24966 consensus_queue.cc:1059] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: INVALID_TERM, Last received: 14.28, Next index: 29, Last known committed idx: 31, Time since last communication: 0.000s
I20260812 08:02:50.495453 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 15 candidate_status { last_received { term: 14 index: 28 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:50.495460 24966 consensus_queue.cc:1243] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Peer 9c1d86a754e645269bc0796059331944 log is divergent from this leader: its last log entry 15.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 31
I20260812 08:02:50.496024 23891 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 LEADER]: Stepping down as leader of term 14
I20260812 08:02:50.496088 23891 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:50.496222 23891 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 14.28, Last appended by leader: 28, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:50.496356 23891 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:02:50.497572 23891 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Refusing update from remote peer 9c1d86a754e645269bc0796059331944: Log matching property violated. Preceding OpId in replica: term: 14 index: 28. Preceding OpId from leader: term: 15 index: 29. (index mismatch)
W20260812 08:02:50.497678 25065 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260812 08:02:50.498040 25170 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20260812 08:02:50.498385 25179 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:50.495194 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.22.155.3:44619)
0812 08:02:50.495881 (+   687us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:50.497985 (+  2104us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:50.498001 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:50.498047 (+    46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:50.498051 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:50.498105 (+    54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:50.498257 (+   152us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51472: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":79,"threads_started":1}
W20260812 08:02:50.501360 25179 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:50.495215 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.22.155.3:44619)
0812 08:02:50.501031 (+  5816us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:50.501036 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:50.501051 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:50.501095 (+    44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:50.501098 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:50.501149 (+    51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:50.501265 (+   116us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51456: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":5696,"thread_start_us":61,"threads_started":1}
W20260812 08:02:51.041333 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:50.520330 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:53036)
0812 08:02:50.520526 (+   196us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:50.520540 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:50.520611 (+    71us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:51.041220 (+520609us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 08:02:51.041574 24009 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
I20260812 08:02:51.521256 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 27 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:51.521463 23434 raft_consensus.cc:1217] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 13.27->[14.28-14.28]   Dedup: 14.28->[]
I20260812 08:02:51.521552 23434 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Rejecting Update request from peer 8811a4670c574f729b04cebef39bc7aa for earlier term 14. Current term is 15. Ops: [14.28-14.28]
I20260812 08:02:51.521797 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 12 candidate_status { last_received { term: 11 index: 26 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:51.521924 23450 raft_consensus.cc:2393] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 12: Already voted for candidate 9c1d86a754e645269bc0796059331944 in this term.
I20260812 08:02:51.522068 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 9 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:51.522177 23428 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 10 because replica has last-logged OpId of term: 9 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 33.
I20260812 08:02:51.522411 25192 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.522552 25192 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
W20260812 08:02:51.523367 23802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:51.523527 25065 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
W20260812 08:02:51.523634 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.090s (SENT)
I20260812 08:02:51.523607 25065 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
W20260812 08:02:51.523767 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.154s (SENT)
I20260812 08:02:51.523809 23807 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:51.524034 25065 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.524037 25193 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.524117 25065 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:02:51.524116 25193 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:51.524989 25193 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:51.525077 25193 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.525122 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:51.525148 25193 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.525252 23903 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 12.
I20260812 08:02:51.525619 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.525663 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:51.525730 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:51.525772 23428 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 15.
I20260812 08:02:51.525801 25193 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.525825 23903 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 15.
I20260812 08:02:51.525882 25193 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260812 08:02:51.525916 25193 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.525950 25193 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:02:51.526043 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.526451 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:51.526597 23807 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.526651 25192 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:51.526767 25192 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260812 08:02:51.526772 25174 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260812 08:02:51.526813 25192 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.526816 25174 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.526856 25192 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:02:51.526862 25174 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:02:51.527472 25193 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:51.527799 25193 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:51.527849 25174 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.528040 25192 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.528154 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:51.528242 25174 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.528541 25192 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.528637 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:51.528249 23891 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:02:51.528755 23896 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in current term 16: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:51.529418 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 10 candidate_status { last_received { term: 9 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:51.529549 23428 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa for term 10 because replica has last-logged OpId of term: 9 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 33.
I20260812 08:02:51.529740 23891 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 13.
I20260812 08:02:51.526602 23903 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 9.
I20260812 08:02:51.530515 23807 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:51.530653 23341 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.530834 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.531186 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:51.531319 23428 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 8811a4670c574f729b04cebef39bc7aa in current term 16: Already voted for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in this term.
I20260812 08:02:51.531898 25174 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 08:02:51.532097 25192 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Leader election won for term 13
I20260812 08:02:51.532186 25192 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:51.532354 25192 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 12.30, Last appended by leader: 30, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:51.533149 25192 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 08:02:51.533242 25192 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9c1d86a754e645269bc0796059331944)
I20260812 08:02:51.533324 25192 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:51.534349 25192 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:51.534866 25192 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 election: Requested vote from peers 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:51.535018 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:51.535120 23903 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:51.536168 23903 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 10.
I20260812 08:02:51.536065 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 12 to 13, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 13 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:51.536577 23341 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.536842 25191 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Leader election won for term 10
W20260812 08:02:51.537022 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51484: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:51.537037 25191 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:51.537251 25191 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 9.34, Last appended by leader: 34, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:51.541249 23210 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 9 to 10, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 10 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:51.546970 23386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49202: Illegal state: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 is not leader of this config: current role FOLLOWER
I20260812 08:02:51.564076 23903 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 12 index: 30. Preceding OpId from leader: term: 13 index: 31. (index mismatch)
I20260812 08:02:51.564584 25191 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260812 08:02:51.565987 23891 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 9 index: 34. Preceding OpId from leader: term: 10 index: 35. (index mismatch)
I20260812 08:02:51.566442 25191 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20260812 08:02:51.608124 23336 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.081s (SENT) [suppressed 3 similar messages]
W20260812 08:02:51.690778 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.076s (SENT)
W20260812 08:02:51.690981 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.081s (SENT)
W20260812 08:02:51.711385 23336 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.092s (SENT)
W20260812 08:02:51.719940 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.097s (SENT)
W20260812 08:02:51.722404 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.092s (SENT)
I20260812 08:02:51.722484 23336 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9; no voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944
I20260812 08:02:51.722654 25193 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260812 08:02:51.727006 23802 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.098s (SENT)
I20260812 08:02:51.727092 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 9c1d86a754e645269bc0796059331944
I20260812 08:02:51.727296 25174 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260812 08:02:51.730614 23802 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.102s (SENT)
W20260812 08:02:51.748071 23336 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.109s (SENT)
I20260812 08:02:51.830067 25192 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:51.830236 25192 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.830720 25192 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.831211 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:51.831385 23903 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 16.
I20260812 08:02:51.831729 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.831904 25192 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260812 08:02:51.831969 25192 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:51.831996 25192 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:02:51.832856 25192 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.833283 25192 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 election: Requested vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:51.833634 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa"
I20260812 08:02:51.833765 23903 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:02:51.834609 23903 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 17.
I20260812 08:02:51.834971 23341 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa; no voters: 
I20260812 08:02:51.835160 25192 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 FOLLOWER]: Leader election won for term 17
I20260812 08:02:51.835286 25192 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 LEADER]: Becoming Leader. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:51.835431 25192 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 15.31, Last appended by leader: 31, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:51.837348 23210 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 reported cstate change: term changed from 15 to 17, leader changed from 9c1d86a754e645269bc0796059331944 (127.22.155.2) to 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1). New cstate: current_term: 17 leader_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:51.865302 23903 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 15 index: 31. Preceding OpId from leader: term: 17 index: 32. (index mismatch)
I20260812 08:02:51.865764 25192 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20260812 08:02:52.027520 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.096s (SENT)
W20260812 08:02:52.052320 23336 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271): Timed out: RequestConsensusVote RPC to 127.22.155.2:33271 timed out after 0.110s (SENT)
W20260812 08:02:52.065245 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:51.557600 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:53040)
0812 08:02:51.557805 (+   205us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:51.557822 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:51.557904 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:52.065145 (+507241us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 08:02:52.065451 24009 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 4 requested bytes
W20260812 08:02:52.075546 23845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51496: Illegal state: replica 8811a4670c574f729b04cebef39bc7aa is not leader of this config: current role FOLLOWER
I20260812 08:02:52.103591 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.103674 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.103591 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.104182 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.104281 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.104478 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.104511 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.104681 23649 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Stepping down as leader of term 12
I20260812 08:02:52.104712 23680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.104761 23649 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:52.104802 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.104905 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.105057 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.104911 23649 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 12.30, Last appended by leader: 30, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:52.105228 23674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.105264 23649 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:02:52.105314 23686 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 LEADER]: Stepping down as leader of term 15
I20260812 08:02:52.105370 23686 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:52.105509 23686 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 15.31, Last appended by leader: 31, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:52.105661 23686 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 15 FOLLOWER]: Advancing to term 17
I20260812 08:02:52.106284 23450 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 9. Current term is 10. Ops: []
I20260812 08:02:52.106302 23656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.106360 23649 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 12 index: 30. Preceding OpId from leader: term: 13 index: 31. (index mismatch)
I20260812 08:02:52.105157 23650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.106550 23670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.106577 23434 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 12. Current term is 13. Ops: []
I20260812 08:02:52.106755 23686 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 15 index: 31. Preceding OpId from leader: term: 17 index: 32. (index mismatch)
I20260812 08:02:52.106784 23658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 17 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.106890 23658 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 17.
I20260812 08:02:52.107020 23428 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 LEADER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 15. Current term is 17. Ops: []
I20260812 08:02:52.107136 25221 consensus_queue.cc:1059] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: INVALID_TERM, Last received: 9.34, Next index: 35, Last known committed idx: 35, Time since last communication: 0.000s
I20260812 08:02:52.107271 23648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.107321 25221 consensus_queue.cc:1243] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [LEADER]: Peer 585f93c1bf1d4cc8b7cf7e588a0584a9 log is divergent from this leader: its last log entry 10.35 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 35
I20260812 08:02:52.107492 25221 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 LEADER]: Stepping down as leader of term 9
I20260812 08:02:52.107579 25221 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9c1d86a754e645269bc0796059331944, State: Running, Role: LEADER
I20260812 08:02:52.107745 23680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 13 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.107759 25221 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 9.34, Last appended by leader: 34, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:52.108069 25221 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:02:52.108672 23680 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 13.
I20260812 08:02:52.108855 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 10 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:52.109135 23685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" candidate_term: 16 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:52.109233 23685 raft_consensus.cc:2368] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 16. Current term is 17.
I20260812 08:02:52.110100 23668 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585f93c1bf1d4cc8b7cf7e588a0584a9 in term 10.
I20260812 08:02:52.110358 23655 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Refusing update from remote peer 585f93c1bf1d4cc8b7cf7e588a0584a9: Log matching property violated. Preceding OpId in replica: term: 9 index: 34. Preceding OpId from leader: term: 10 index: 35. (index mismatch)
I20260812 08:02:52.115252 23891 raft_consensus.cc:1240] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Rejecting Update request from peer 9c1d86a754e645269bc0796059331944 for earlier term 12. Current term is 13. Ops: []
W20260812 08:02:52.115335 25222 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:52.114174 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.22.155.2:33271)
0812 08:02:52.114428 (+   254us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:52.114433 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:52.114446 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:52.114489 (+    43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:52.114493 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:52.114538 (+    45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:52.115222 (+   684us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53040: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
W20260812 08:02:52.641319 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:52.139950 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:49234)
0812 08:02:52.140090 (+   140us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:52.140104 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:52.140182 (+    78us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:52.641206 (+501024us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 08:02:53.140450 25174 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.140597 25174 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:53.141141 25174 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.141237 25174 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.141299 25174 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.141718 25174 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.141808 25174 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.141868 25174 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.142201 25174 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.142273 25139 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:53.142382 25139 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:53.142725 25139 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619), 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399)
W20260812 08:02:53.143211 23566 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:53.143618 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:53.143716 23655 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 17.
I20260812 08:02:53.143877 25139 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:53.143936 25139 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.144213 25139 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:53.144271 25139 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:53.144318 25139 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.144556 25139 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 8811a4670c574f729b04cebef39bc7aa (127.22.155.3:44619)
I20260812 08:02:53.144981 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:53.145002 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:53.145116 23668 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 10.
I20260812 08:02:53.145134 25174 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260812 08:02:53.145186 25174 raft_consensus.cc:493] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.145228 25174 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:02:53.145292 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944" is_pre_election: true
I20260812 08:02:53.145363 23655 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 13.
W20260812 08:02:53.146361 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:52.644224 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.2:33271 (local address 127.0.0.1:53054)
0812 08:02:52.644414 (+   190us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:52.644428 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:52.644511 (+    83us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:53.146257 (+501746us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 25 requested bytes
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 08:02:53.146416 23891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 18 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:53.146644 23896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
W20260812 08:02:53.146725 25229 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:53.143388 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.22.155.2:33271)
0812 08:02:53.145960 (+  2572us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:53.145965 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:53.145980 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:53.146025 (+    45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:53.146028 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:53.146077 (+    49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:53.146628 (+   551us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53054: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":2489,"thread_start_us":58,"threads_started":1}
I20260812 08:02:53.146740 23896 raft_consensus.cc:2410] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 11 because replica has last-logged OpId of term: 10 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 34.
I20260812 08:02:53.146929 23903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "8811a4670c574f729b04cebef39bc7aa" is_pre_election: true
I20260812 08:02:53.147027 23903 raft_consensus.cc:2410] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 for term 14 because replica has last-logged OpId of term: 13 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 30.
W20260812 08:02:53.147140 25229 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:53.145593 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.22.155.2:33271)
0812 08:02:53.146875 (+  1282us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:53.146879 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:53.146890 (+    11us) server_negotiation.cc:408] Connection header received
0812 08:02:53.146925 (+    35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:53.146928 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:53.146971 (+    43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:53.147071 (+   100us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53036: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":55,"threads_started":1}
I20260812 08:02:53.147491 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
W20260812 08:02:53.147601 24009 meta_cache.cc:302] tablet e82eb9371afc404ca426b35866b8fc52: replica 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.2:33271: received 0 of 25 requested bytes
I20260812 08:02:53.147636 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:53.146422 25174 raft_consensus.cc:515] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:53.148228 25174 leader_election.cc:290] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.148309 25174 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260812 08:02:53.148347 25174 raft_consensus.cc:493] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.148376 23891 raft_consensus.cc:2393] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c1d86a754e645269bc0796059331944 in current term 18: Already voted for candidate 8811a4670c574f729b04cebef39bc7aa in this term.
I20260812 08:02:53.148381 25174 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:02:53.149309 25232 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 08:02:53.149428 25232 raft_consensus.cc:493] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Starting leader election (detected failure of leader 585f93c1bf1d4cc8b7cf7e588a0584a9)
I20260812 08:02:53.149534 25232 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:53.149582 25174 raft_consensus.cc:515] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.149911 25174 leader_election.cc:290] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.150274 23655 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:53.150372 23655 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:02:53.150884 25232 raft_consensus.cc:515] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.151410 23655 raft_consensus.cc:2468] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 18.
I20260812 08:02:53.151525 25232 leader_election.cc:290] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 election: Requested vote from peers 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399), 9c1d86a754e645269bc0796059331944 (127.22.155.2:33271)
I20260812 08:02:53.151592 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:53.151680 23668 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:02:53.152518 23668 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14.
I20260812 08:02:53.153009 23802 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:53.153054 23668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "9c1d86a754e645269bc0796059331944"
I20260812 08:02:53.153149 23668 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:53.153196 23802 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:53.153401 25174 raft_consensus.cc:2804] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 18 FOLLOWER]: Leader election won for term 18
I20260812 08:02:53.153573 25226 raft_consensus.cc:2804] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 14 FOLLOWER]: Leader election won for term 14
I20260812 08:02:53.153695 25226 raft_consensus.cc:697] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [term 14 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:53.153829 25226 consensus_queue.cc:237] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 13.33, Last appended by leader: 33, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.154004 23668 raft_consensus.cc:2468] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 11.
I20260812 08:02:53.154552 23802 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8811a4670c574f729b04cebef39bc7aa, 9c1d86a754e645269bc0796059331944; no voters: 
I20260812 08:02:53.154706 25174 raft_consensus.cc:697] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [term 18 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:53.154881 25174 consensus_queue.cc:237] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 17.33, Last appended by leader: 33, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:53.154919 25232 raft_consensus.cc:2804] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 11 FOLLOWER]: Leader election won for term 11
I20260812 08:02:53.155123 25232 raft_consensus.cc:697] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 11 LEADER]: Becoming Leader. State: Replica: 8811a4670c574f729b04cebef39bc7aa, State: Running, Role: LEADER
I20260812 08:02:53.155341 25232 consensus_queue.cc:237] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 10.35, Last appended by leader: 35, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:53.156647 23210 catalog_manager.cc:5719] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 17 to 18, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 18 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:53.156916 23210 catalog_manager.cc:5719] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 13 to 14, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 14 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:53.160847 23668 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 9 index: 34. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
I20260812 08:02:53.161275 25238 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 34, Time since last communication: 0.000s
I20260812 08:02:53.163630 23210 catalog_manager.cc:5719] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa reported cstate change: term changed from 10 to 11, leader changed from 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1) to 8811a4670c574f729b04cebef39bc7aa (127.22.155.3). New cstate: current_term: 11 leader_uuid: "8811a4670c574f729b04cebef39bc7aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:53.181787 23668 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 18 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 15 index: 31. Preceding OpId from leader: term: 18 index: 34. (index mismatch)
I20260812 08:02:53.181777 23655 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 12 index: 30. Preceding OpId from leader: term: 14 index: 34. (index mismatch)
I20260812 08:02:53.182343 25238 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 30, Time since last communication: 0.000s
I20260812 08:02:53.182430 25232 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 31, Time since last communication: 0.000s
W20260812 08:02:53.219552 23570 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.076s (SENT) [suppressed 9 similar messages]
W20260812 08:02:53.227916 23807 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.084s (SENT) [suppressed 7 similar messages]
W20260812 08:02:53.296411 23570 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.076s (SENT)
I20260812 08:02:53.296532 23570 leader_election.cc:304] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:53.296746 25139 raft_consensus.cc:2749] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260812 08:02:53.311890 23570 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.083s (SENT)
I20260812 08:02:53.312103 23570 leader_election.cc:304] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:53.312307 25139 raft_consensus.cc:2749] T 88f4b4af90a54b149dd1c350d0040d7c P 9c1d86a754e645269bc0796059331944 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260812 08:02:53.312664 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.084s (SENT)
W20260812 08:02:53.313992 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.085s (SENT)
W20260812 08:02:53.315479 23570 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.085s (SENT)
I20260812 08:02:53.315546 23570 leader_election.cc:304] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1d86a754e645269bc0796059331944; no voters: 585f93c1bf1d4cc8b7cf7e588a0584a9, 8811a4670c574f729b04cebef39bc7aa
I20260812 08:02:53.315692 25139 raft_consensus.cc:2749] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260812 08:02:53.336524 23807 leader_election.cc:336] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.092s (SENT)
W20260812 08:02:53.348354 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.103s (SENT)
W20260812 08:02:53.351786 23807 leader_election.cc:336] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.101s (SENT)
W20260812 08:02:53.371249 23807 leader_election.cc:336] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Timed out: RequestConsensusVote RPC to 127.22.155.1:46399 timed out after 0.110s (SENT)
W20260812 08:02:53.701284 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:53.167434 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:49244)
0812 08:02:53.167597 (+   163us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:53.167610 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:53.167707 (+    97us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:53.701172 (+533465us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 08:02:53.707075 23618 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53062: Illegal state: replica 9c1d86a754e645269bc0796059331944 is not leader of this config: current role FOLLOWER
W20260812 08:02:54.241400 24013 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:53.719831 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.1:46399 (local address 127.0.0.1:49248)
0812 08:02:53.719992 (+   161us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:53.720006 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:53.720074 (+    68us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:54.241243 (+521169us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 08:02:54.241626 24011 meta_cache.cc:302] tablet 21ed101d57de4eb983765b92682e2b5f: replica 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399) has failed: Timed out: Client connection negotiation failed: client connection to 127.22.155.1:46399: received 0 of 4 requested bytes
I20260812 08:02:54.253741 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254012 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254102 23428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 18 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254247 23434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 14 candidate_status { last_received { term: 12 index: 30 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254344 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 18 candidate_status { last_received { term: 15 index: 31 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254439 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.254575 23449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254642 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.254835 23436 raft_consensus.cc:3055] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Stepping down as leader of term 13
I20260812 08:02:54.254881 23436 raft_consensus.cc:740] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:54.255007 23436 consensus_queue.cc:260] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 13.33, Last appended by leader: 33, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:54.255141 23436 raft_consensus.cc:3060] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 13 FOLLOWER]: Advancing to term 14
W20260812 08:02:54.255501 23336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260812 08:02:54.256269 23436 raft_consensus.cc:1275] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 13 index: 33. Preceding OpId from leader: term: 14 index: 34. (index mismatch)
I20260812 08:02:54.256493 23430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.256596 23441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "9c1d86a754e645269bc0796059331944" candidate_term: 11 candidate_status { last_received { term: 9 index: 34 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.256891 23431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.257041 23440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.257138 23450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.257220 23435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.257270 25192 consensus_queue.cc:1048] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260812 08:02:54.257498 23430 raft_consensus.cc:2468] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14.
I20260812 08:02:54.257567 25192 consensus_queue.cc:1048] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20260812 08:02:54.257762 23434 raft_consensus.cc:3055] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 LEADER]: Stepping down as leader of term 17
I20260812 08:02:54.257808 23434 raft_consensus.cc:740] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:54.257915 23434 consensus_queue.cc:260] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 17.33, Last appended by leader: 33, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } } peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } }
I20260812 08:02:54.258033 23434 raft_consensus.cc:3060] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:02:54.254587 23447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e82eb9371afc404ca426b35866b8fc52" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 11 candidate_status { last_received { term: 10 index: 35 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.257234 23429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21ed101d57de4eb983765b92682e2b5f" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 18 candidate_status { last_received { term: 17 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9"
I20260812 08:02:54.258215 25174 consensus_queue.cc:1048] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260812 08:02:54.259045 23434 raft_consensus.cc:1275] T 21ed101d57de4eb983765b92682e2b5f P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 18 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 17 index: 33. Preceding OpId from leader: term: 18 index: 34. (index mismatch)
I20260812 08:02:54.259071 23438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
I20260812 08:02:54.259166 23438 raft_consensus.cc:2376] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14. Re-sending same reply.
I20260812 08:02:54.259394 23428 raft_consensus.cc:3055] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 LEADER]: Stepping down as leader of term 10
I20260812 08:02:54.259461 23428 raft_consensus.cc:740] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 585f93c1bf1d4cc8b7cf7e588a0584a9, State: Running, Role: LEADER
I20260812 08:02:54.259595 23428 consensus_queue.cc:260] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 10.35, Last appended by leader: 35, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 46399 } } peers { permanent_uuid: "9c1d86a754e645269bc0796059331944" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 33271 } } peers { permanent_uuid: "8811a4670c574f729b04cebef39bc7aa" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44619 } }
I20260812 08:02:54.259809 23428 raft_consensus.cc:3060] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:02:54.259939 23420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88f4b4af90a54b149dd1c350d0040d7c" candidate_uuid: "8811a4670c574f729b04cebef39bc7aa" candidate_term: 14 candidate_status { last_received { term: 13 index: 33 } } ignore_live_leader: false dest_uuid: "585f93c1bf1d4cc8b7cf7e588a0584a9" is_pre_election: true
W20260812 08:02:54.259984 25293 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:54.255255 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.22.155.1:46399)
0812 08:02:54.259342 (+  4087us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:54.259347 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:54.259362 (+    15us) server_negotiation.cc:408] Connection header received
0812 08:02:54.259400 (+    38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:54.259403 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:54.259449 (+    46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:54.259567 (+   118us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49234: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":4006,"thread_start_us":60,"threads_started":1}
I20260812 08:02:54.260027 23420 raft_consensus.cc:2376] T 88f4b4af90a54b149dd1c350d0040d7c P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8811a4670c574f729b04cebef39bc7aa in term 14. Re-sending same reply.
W20260812 08:02:54.256644 25217 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:54.253828 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.22.155.1:46399)
0812 08:02:54.256293 (+  2465us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:54.256298 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:54.256311 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:54.256354 (+    43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:54.256357 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:54.256415 (+    58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:54.256535 (+   120us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49244: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2361}
W20260812 08:02:54.260448 25293 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:54.255646 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.22.155.1:46399)
0812 08:02:54.260130 (+  4484us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:54.260134 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:54.260147 (+    13us) server_negotiation.cc:408] Connection header received
0812 08:02:54.260186 (+    39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:54.260189 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:54.260232 (+    43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:54.260339 (+   107us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49248: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":4426,"thread_start_us":94,"threads_started":1}
I20260812 08:02:54.260747 23428 raft_consensus.cc:1275] T e82eb9371afc404ca426b35866b8fc52 P 585f93c1bf1d4cc8b7cf7e588a0584a9 [term 11 FOLLOWER]: Refusing update from remote peer 8811a4670c574f729b04cebef39bc7aa: Log matching property violated. Preceding OpId in replica: term: 10 index: 35. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
I20260812 08:02:54.261952 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 23312
I20260812 08:02:54.265832 23891 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa [term 11 LEADER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 10. Current term is 11. Ops: []
I20260812 08:02:54.271495 23668 raft_consensus.cc:1217] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 9.34->[10.35-10.35]   Dedup: 10.35->[]
I20260812 08:02:54.271629 23668 raft_consensus.cc:1240] T e82eb9371afc404ca426b35866b8fc52 P 9c1d86a754e645269bc0796059331944 [term 11 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 10. Current term is 11. Ops: [10.35-10.35]
I20260812 08:02:54.272557 23655 raft_consensus.cc:1217] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 18 FOLLOWER]: Deduplicated request from leader. Original: 15.31->[17.32-17.33]   Dedup: 17.33->[]
I20260812 08:02:54.272667 23655 raft_consensus.cc:1240] T 21ed101d57de4eb983765b92682e2b5f P 9c1d86a754e645269bc0796059331944 [term 18 FOLLOWER]: Rejecting Update request from peer 585f93c1bf1d4cc8b7cf7e588a0584a9 for earlier term 17. Current term is 18. Ops: [17.32-17.33]
W20260812 08:02:54.279064 23568 connection.cc:573] server connection from 127.22.155.1:47399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:54.279080 23802 connection.cc:573] server connection from 127.22.155.1:54813 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:54.279345 23807 connection.cc:573] client connection to 127.22.155.1:46399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:54.279333 23185 connection.cc:573] server connection from 127.22.155.1:44243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:02:54.279633 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 23541
W20260812 08:02:54.288132 23807 consensus_peers.cc:597] T e82eb9371afc404ca426b35866b8fc52 P 8811a4670c574f729b04cebef39bc7aa -> Peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Couldn't send request to peer 585f93c1bf1d4cc8b7cf7e588a0584a9. Status: Network error: Client connection negotiation failed: client connection to 127.22.155.1:46399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:54.288244 23807 consensus_peers.cc:597] T 88f4b4af90a54b149dd1c350d0040d7c P 8811a4670c574f729b04cebef39bc7aa -> Peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Couldn't send request to peer 585f93c1bf1d4cc8b7cf7e588a0584a9. Status: Network error: Client connection negotiation failed: client connection to 127.22.155.1:46399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:54.288283 23807 consensus_peers.cc:597] T 21ed101d57de4eb983765b92682e2b5f P 8811a4670c574f729b04cebef39bc7aa -> Peer 585f93c1bf1d4cc8b7cf7e588a0584a9 (127.22.155.1:46399): Couldn't send request to peer 585f93c1bf1d4cc8b7cf7e588a0584a9. Status: Network error: Client connection negotiation failed: client connection to 127.22.155.1:46399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:54.294905 23802 connection.cc:573] client connection to 127.22.155.2:33271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:02:54.295423 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 23778
I20260812 08:02:54.305729 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 23163
2026-08-12T08:02:54Z chronyd exiting
[       OK ] ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer (38180 ms)
[ RUN      ] ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause
2026-08-12T08:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:54Z Disabled control of system clock
I20260812 08:02:54.343031 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:44601
--webserver_interface=127.22.155.62
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:44601
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
W20260812 08:02:54.449795 25301 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:54.450042 25301 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:54.450085 25301 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:54.453471 25301 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:54.453547 25301 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:54.453567 25301 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:54.453588 25301 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:54.453605 25301 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:54.457600 25301 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:44601
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.155.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:54.458720 25301 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:54.459831 25301 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:54.465875 25307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.465890 25308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.465875 25310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:54.465966 25301 server_base.cc:1061] running on GCE node
I20260812 08:02:54.466768 25301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:54.467828 25301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:54.469026 25301 hybrid_clock.cc:648] HybridClock initialized: now 1786521774469007 us; error 37 us; skew 500 ppm
I20260812 08:02:54.470959 25301 webserver.cc:533] Webserver started at http://127.22.155.62:43373/ using document root <none> and password file <none>
I20260812 08:02:54.471577 25301 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:54.471665 25301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:54.471915 25301 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:54.473579 25301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data/instance:
uuid: "f36ad7d7c15a4cda9781b7ea9d362a12"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.474076 25301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal/instance:
uuid: "f36ad7d7c15a4cda9781b7ea9d362a12"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.477415 25301 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:54.479691 25319 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.480654 25301 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:54.480805 25301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal
uuid: "f36ad7d7c15a4cda9781b7ea9d362a12"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.480932 25301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:54.499951 25301 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:54.500708 25301 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:54.500921 25301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:54.508646 25409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.62:44601 every 8 connection(s)
I20260812 08:02:54.508651 25301 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.62:44601
I20260812 08:02:54.509756 25301 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:54.512535 25410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:54.517464 25410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12: Bootstrap starting.
I20260812 08:02:54.518728 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 25301
I20260812 08:02:54.518839 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/master-0/wal/instance
I20260812 08:02:54.520008 25410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:54.520862 25410 log.cc:826] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:54.522604 25410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12: No bootstrap required, opened a new log
I20260812 08:02:54.525166 25410 raft_consensus.cc:359] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } }
I20260812 08:02:54.525363 25410 raft_consensus.cc:385] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:54.525447 25410 raft_consensus.cc:740] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36ad7d7c15a4cda9781b7ea9d362a12, State: Initialized, Role: FOLLOWER
I20260812 08:02:54.525929 25410 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } }
I20260812 08:02:54.526094 25410 raft_consensus.cc:399] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:02:54.526162 25410 raft_consensus.cc:493] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:02:54.526273 25410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:54.527158 25410 raft_consensus.cc:515] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } }
I20260812 08:02:54.527487 25410 leader_election.cc:304] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36ad7d7c15a4cda9781b7ea9d362a12; no voters: 
I20260812 08:02:54.527810 25410 leader_election.cc:290] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:02:54.527923 25418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:54.528144 25418 raft_consensus.cc:697] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [term 1 LEADER]: Becoming Leader. State: Replica: f36ad7d7c15a4cda9781b7ea9d362a12, State: Running, Role: LEADER
I20260812 08:02:54.528491 25418 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } }
I20260812 08:02:54.528882 25410 sys_catalog.cc:565] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:54.531095 25419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } } }
I20260812 08:02:54.531209 25419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:54.531788 25420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36ad7d7c15a4cda9781b7ea9d362a12. Latest consensus state: current_term: 1 leader_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36ad7d7c15a4cda9781b7ea9d362a12" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 44601 } } }
I20260812 08:02:54.531889 25420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:54.532015 25444 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:54.534992 25444 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:54.540580 25444 catalog_manager.cc:1383] Generated new cluster ID: 0a5112a27ed54f68b4f50788e9bccebe
I20260812 08:02:54.540694 25444 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:54.551103 25444 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:54.552178 25444 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:54.567870 25444 catalog_manager.cc:6092] T 00000000000000000000000000000000 P f36ad7d7c15a4cda9781b7ea9d362a12: Generated new TSK 0
I20260812 08:02:54.568543 25444 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:54.572127 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.1:0
--local_ip_for_outbound_sockets=127.22.155.1
--webserver_interface=127.22.155.1
--webserver_port=0
--tserver_master_addrs=127.22.155.62:44601
--builtin_ntp_servers=127.22.155.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
W20260812 08:02:54.685266 25455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:54.685487 25455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:54.685583 25455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:54.689102 25455 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:54.689172 25455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:54.689249 25455 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:54.689363 25455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.1
I20260812 08:02:54.693437 25455 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.155.1
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:44601
--never_fsync=true
--heap_profile_path=/tmp/kudu.25455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:54.694586 25455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:54.695698 25455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:54.698771 25455 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:54.702510 25464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.702556 25466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.702556 25470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:54.702847 25455 server_base.cc:1061] running on GCE node
I20260812 08:02:54.703256 25455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:54.703882 25455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:54.705008 25455 hybrid_clock.cc:648] HybridClock initialized: now 1786521774704986 us; error 32 us; skew 500 ppm
I20260812 08:02:54.707116 25455 webserver.cc:533] Webserver started at http://127.22.155.1:38685/ using document root <none> and password file <none>
I20260812 08:02:54.707808 25455 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:54.707876 25455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:54.708074 25455 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:54.710215 25455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data/instance:
uuid: "83539aa895e1497ab4b0a263cd4797cc"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.710781 25455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance:
uuid: "83539aa895e1497ab4b0a263cd4797cc"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.714855 25455 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:54.717110 25481 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.718232 25455 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:54.718357 25455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal
uuid: "83539aa895e1497ab4b0a263cd4797cc"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.718441 25455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:54.739290 25455 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:54.739929 25455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:54.740082 25455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:54.741706 25455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:54.741794 25455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.741876 25455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:54.741921 25455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.752609 25455 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.1:44059
I20260812 08:02:54.752635 25684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.1:44059 every 8 connection(s)
I20260812 08:02:54.753767 25455 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:54.755213 25685 heartbeater.cc:344] Connected to a master server at 127.22.155.62:44601
I20260812 08:02:54.755391 25685 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:54.755978 25685 heartbeater.cc:507] Master 127.22.155.62:44601 requested a full tablet report, sending...
I20260812 08:02:54.757537 25348 ts_manager.cc:194] Registered new tserver with Master: 83539aa895e1497ab4b0a263cd4797cc (127.22.155.1:44059)
I20260812 08:02:54.759002 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 25455
I20260812 08:02:54.759044 25348 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.1:52825
I20260812 08:02:54.759125 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance
I20260812 08:02:54.765182 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.2:0
--local_ip_for_outbound_sockets=127.22.155.2
--webserver_interface=127.22.155.2
--webserver_port=0
--tserver_master_addrs=127.22.155.62:44601
--builtin_ntp_servers=127.22.155.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
I20260812 08:02:54.786559 25685 heartbeater.cc:499] Master 127.22.155.62:44601 was elected leader, sending a full tablet report...
W20260812 08:02:54.878026 25697 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:54.878268 25697 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:54.878347 25697 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:54.881857 25697 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:54.881958 25697 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:54.882050 25697 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:54.882144 25697 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.2
I20260812 08:02:54.885957 25697 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.155.2
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:44601
--never_fsync=true
--heap_profile_path=/tmp/kudu.25697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.2

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:54.887017 25697 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:54.888123 25697 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:54.890959 25697 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:54.894464 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.894511 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:54.894529 25707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:54.894816 25697 server_base.cc:1061] running on GCE node
I20260812 08:02:54.895232 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:54.895833 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:54.897022 25697 hybrid_clock.cc:648] HybridClock initialized: now 1786521774897007 us; error 30 us; skew 500 ppm
I20260812 08:02:54.898787 25697 webserver.cc:533] Webserver started at http://127.22.155.2:37689/ using document root <none> and password file <none>
I20260812 08:02:54.899358 25697 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:54.899415 25697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:54.899616 25697 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:54.901276 25697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data/instance:
uuid: "164745ab533a4fd48893689e86e7c197"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.902113 25697 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal/instance:
uuid: "164745ab533a4fd48893689e86e7c197"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.905670 25697 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:02:54.907765 25718 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.908666 25697 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:54.908802 25697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal
uuid: "164745ab533a4fd48893689e86e7c197"
format_stamp: "Formatted at 2026-08-12 08:02:54 on dist-test-slave-ffrd"
I20260812 08:02:54.908905 25697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:54.930562 25697 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:54.931357 25697 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:54.931557 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:54.933259 25697 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:54.933332 25697 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.933403 25697 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:54.933454 25697 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:54.946098 25697 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.2:37515
I20260812 08:02:54.946177 25905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.2:37515 every 8 connection(s)
I20260812 08:02:54.947060 25697 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/data/info.pb
I20260812 08:02:54.948503 25906 heartbeater.cc:344] Connected to a master server at 127.22.155.62:44601
I20260812 08:02:54.948731 25906 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:54.949329 25906 heartbeater.cc:507] Master 127.22.155.62:44601 requested a full tablet report, sending...
I20260812 08:02:54.950284 25348 ts_manager.cc:194] Registered new tserver with Master: 164745ab533a4fd48893689e86e7c197 (127.22.155.2:37515)
I20260812 08:02:54.950968 25348 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.2:57249
I20260812 08:02:54.951998 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 25697
I20260812 08:02:54.952087 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-1/wal/instance
I20260812 08:02:54.957619 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.3:0
--local_ip_for_outbound_sockets=127.22.155.3
--webserver_interface=127.22.155.3
--webserver_port=0
--tserver_master_addrs=127.22.155.62:44601
--builtin_ntp_servers=127.22.155.20:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
I20260812 08:02:54.977689 25906 heartbeater.cc:499] Master 127.22.155.62:44601 was elected leader, sending a full tablet report...
W20260812 08:02:55.071301 25910 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:55.071519 25910 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:55.071570 25910 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:55.075006 25910 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:55.075075 25910 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:55.075117 25910 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:55.075160 25910 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.3
I20260812 08:02:55.079075 25910 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.155.3
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:44601
--never_fsync=true
--heap_profile_path=/tmp/kudu.25910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.3

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:55.080142 25910 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:55.081336 25910 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:55.084149 25910 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:55.088114 25920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:55.088160 25925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:55.088186 25923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:55.088557 25910 server_base.cc:1061] running on GCE node
I20260812 08:02:55.089000 25910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:55.089576 25910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:55.090772 25910 hybrid_clock.cc:648] HybridClock initialized: now 1786521775090753 us; error 30 us; skew 500 ppm
I20260812 08:02:55.092837 25910 webserver.cc:533] Webserver started at http://127.22.155.3:38741/ using document root <none> and password file <none>
I20260812 08:02:55.093500 25910 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:55.093590 25910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:55.093845 25910 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:55.095422 25910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data/instance:
uuid: "01e4ae4d92be471383bd23690e34b55b"
format_stamp: "Formatted at 2026-08-12 08:02:55 on dist-test-slave-ffrd"
I20260812 08:02:55.095958 25910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal/instance:
uuid: "01e4ae4d92be471383bd23690e34b55b"
format_stamp: "Formatted at 2026-08-12 08:02:55 on dist-test-slave-ffrd"
I20260812 08:02:55.099390 25910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 08:02:55.101703 25933 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:55.102785 25910 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:55.102929 25910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal
uuid: "01e4ae4d92be471383bd23690e34b55b"
format_stamp: "Formatted at 2026-08-12 08:02:55 on dist-test-slave-ffrd"
I20260812 08:02:55.103040 25910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:55.112586 25910 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:55.113322 25910 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:55.113528 25910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:55.115278 25910 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:55.115343 25910 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:55.115525 25910 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:55.115562 25910 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:55.127660 25910 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.3:44441
I20260812 08:02:55.128927 25910 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/data/info.pb
I20260812 08:02:55.130250 26122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.3:44441 every 8 connection(s)
I20260812 08:02:55.131776 26123 heartbeater.cc:344] Connected to a master server at 127.22.155.62:44601
I20260812 08:02:55.131963 26123 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:55.132397 26123 heartbeater.cc:507] Master 127.22.155.62:44601 requested a full tablet report, sending...
I20260812 08:02:55.133399 25348 ts_manager.cc:194] Registered new tserver with Master: 01e4ae4d92be471383bd23690e34b55b (127.22.155.3:44441)
I20260812 08:02:55.133908 25348 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.3:46559
I20260812 08:02:55.134498 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 25910
I20260812 08:02:55.134608 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause.1786521736143602-23148-0/minicluster-data/ts-2/wal/instance
I20260812 08:02:55.144539 23148 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:55.156617 25348 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57722:
name: "kudu_system.kudu_transactions"
schema {
  columns {
    name: "txn_id"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "entry_type"
    type: INT8
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "identifier"
    type: STRING
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "metadata"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001d\000\000\000\000\000\000\000"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "txn_id"
    }
  }
}
table_type: TXN_STATUS_TABLE
W20260812 08:02:55.159390 25348 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:02:55.161103 26123 heartbeater.cc:499] Master 127.22.155.62:44601 was elected leader, sending a full tablet report...
I20260812 08:02:55.176699 26006 tablet_service.cc:1511] Processing CreateTablet for tablet 69291db43d384d4c96bdafe659f125ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5673d6d80844e18847f935aaf06411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 100
I20260812 08:02:55.176726 25788 tablet_service.cc:1511] Processing CreateTablet for tablet 69291db43d384d4c96bdafe659f125ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5673d6d80844e18847f935aaf06411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 100
I20260812 08:02:55.178004 25788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69291db43d384d4c96bdafe659f125ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:55.178015 26006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69291db43d384d4c96bdafe659f125ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:55.178149 25572 tablet_service.cc:1511] Processing CreateTablet for tablet 69291db43d384d4c96bdafe659f125ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a5673d6d80844e18847f935aaf06411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 100
I20260812 08:02:55.179275 25572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69291db43d384d4c96bdafe659f125ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:55.185441 26148 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Bootstrap starting.
I20260812 08:02:55.186360 26149 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Bootstrap starting.
I20260812 08:02:55.186772 26152 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Bootstrap starting.
I20260812 08:02:55.187796 26148 tablet_bootstrap.cc:654] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:55.188470 26149 tablet_bootstrap.cc:654] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:55.189951 26152 tablet_bootstrap.cc:654] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:55.190822 26152 log.cc:826] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:55.191187 26148 log.cc:826] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:55.192025 26149 log.cc:826] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:55.192785 26152 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: No bootstrap required, opened a new log
I20260812 08:02:55.193022 26152 ts_tablet_manager.cc:1403] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:02:55.193665 26149 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: No bootstrap required, opened a new log
I20260812 08:02:55.193737 26148 tablet_bootstrap.cc:492] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: No bootstrap required, opened a new log
I20260812 08:02:55.193868 26149 ts_tablet_manager.cc:1403] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20260812 08:02:55.193915 26148 ts_tablet_manager.cc:1403] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20260812 08:02:55.196979 26152 raft_consensus.cc:359] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.197198 26152 raft_consensus.cc:385] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:55.197265 26152 raft_consensus.cc:740] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e4ae4d92be471383bd23690e34b55b, State: Initialized, Role: FOLLOWER
I20260812 08:02:55.197547 26148 raft_consensus.cc:359] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.197527 26149 raft_consensus.cc:359] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.197726 26148 raft_consensus.cc:385] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:55.197741 26149 raft_consensus.cc:385] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:55.197778 26148 raft_consensus.cc:740] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164745ab533a4fd48893689e86e7c197, State: Initialized, Role: FOLLOWER
I20260812 08:02:55.197798 26149 raft_consensus.cc:740] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83539aa895e1497ab4b0a263cd4797cc, State: Initialized, Role: FOLLOWER
I20260812 08:02:55.198140 26148 consensus_queue.cc:260] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.198246 26149 consensus_queue.cc:260] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.199112 26148 ts_tablet_manager.cc:1434] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:02:55.199254 26149 ts_tablet_manager.cc:1434] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:02:55.200512 26161 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:55.200661 26161 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 08:02:55.200991 26160 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:55.201116 26160 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 08:02:55.197737 26152 consensus_queue.cc:260] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.203174 26152 ts_tablet_manager.cc:1434] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260812 08:02:55.203780 26167 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:55.203927 26167 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 08:02:55.211695 26160 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:55.211989 26160 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.213246 26160 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e4ae4d92be471383bd23690e34b55b (127.22.155.3:44441), 83539aa895e1497ab4b0a263cd4797cc (127.22.155.1:44059)
I20260812 08:02:55.216012 26161 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:55.216192 26161 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.217244 26167 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:55.217411 26167 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.217664 26012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "164745ab533a4fd48893689e86e7c197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e4ae4d92be471383bd23690e34b55b" is_pre_election: true
I20260812 08:02:55.217834 26012 raft_consensus.cc:2468] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 164745ab533a4fd48893689e86e7c197 in term 0.
I20260812 08:02:55.218412 26167 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 164745ab533a4fd48893689e86e7c197 (127.22.155.2:37515), 83539aa895e1497ab4b0a263cd4797cc (127.22.155.1:44059)
I20260812 08:02:55.219300 25601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "164745ab533a4fd48893689e86e7c197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83539aa895e1497ab4b0a263cd4797cc" is_pre_election: true
I20260812 08:02:55.219501 25601 raft_consensus.cc:2468] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 164745ab533a4fd48893689e86e7c197 in term 0.
I20260812 08:02:55.219822 26161 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 164745ab533a4fd48893689e86e7c197 (127.22.155.2:37515), 01e4ae4d92be471383bd23690e34b55b (127.22.155.3:44441)
I20260812 08:02:55.219877 25723 leader_election.cc:304] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e4ae4d92be471383bd23690e34b55b, 164745ab533a4fd48893689e86e7c197; no voters: 
I20260812 08:02:55.220152 26160 raft_consensus.cc:2804] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:55.220239 26160 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:55.220288 26160 raft_consensus.cc:3060] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:55.221596 26160 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.222318 26012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "164745ab533a4fd48893689e86e7c197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e4ae4d92be471383bd23690e34b55b"
I20260812 08:02:55.222438 26012 raft_consensus.cc:3060] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:55.222613 25822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "83539aa895e1497ab4b0a263cd4797cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164745ab533a4fd48893689e86e7c197" is_pre_election: true
I20260812 08:02:55.222841 25822 raft_consensus.cc:2393] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83539aa895e1497ab4b0a263cd4797cc in current term 1: Already voted for candidate 164745ab533a4fd48893689e86e7c197 in this term.
I20260812 08:02:55.223590 26012 raft_consensus.cc:2468] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 164745ab533a4fd48893689e86e7c197 in term 1.
I20260812 08:02:55.224000 25723 leader_election.cc:304] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e4ae4d92be471383bd23690e34b55b, 164745ab533a4fd48893689e86e7c197; no voters: 
I20260812 08:02:55.224668 25601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "164745ab533a4fd48893689e86e7c197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83539aa895e1497ab4b0a263cd4797cc"
I20260812 08:02:55.224817 25601 raft_consensus.cc:3060] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:55.225435 25822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "01e4ae4d92be471383bd23690e34b55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164745ab533a4fd48893689e86e7c197" is_pre_election: true
I20260812 08:02:55.225587 25822 raft_consensus.cc:2393] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01e4ae4d92be471383bd23690e34b55b in current term 1: Already voted for candidate 164745ab533a4fd48893689e86e7c197 in this term.
I20260812 08:02:55.225921 25601 raft_consensus.cc:2468] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 164745ab533a4fd48893689e86e7c197 in term 1.
I20260812 08:02:55.226125 26160 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [CANDIDATE]: Term 1 election: Requested vote from peers 01e4ae4d92be471383bd23690e34b55b (127.22.155.3:44441), 83539aa895e1497ab4b0a263cd4797cc (127.22.155.1:44059)
I20260812 08:02:55.226253 26160 raft_consensus.cc:2804] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:55.226348 26160 raft_consensus.cc:697] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [term 1 LEADER]: Becoming Leader. State: Replica: 164745ab533a4fd48893689e86e7c197, State: Running, Role: LEADER
I20260812 08:02:55.226577 25601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "01e4ae4d92be471383bd23690e34b55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83539aa895e1497ab4b0a263cd4797cc" is_pre_election: true
I20260812 08:02:55.226630 26160 consensus_queue.cc:237] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:55.226750 25601 raft_consensus.cc:2393] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01e4ae4d92be471383bd23690e34b55b in current term 1: Already voted for candidate 164745ab533a4fd48893689e86e7c197 in this term.
I20260812 08:02:55.227183 25936 leader_election.cc:304] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e4ae4d92be471383bd23690e34b55b; no voters: 164745ab533a4fd48893689e86e7c197, 83539aa895e1497ab4b0a263cd4797cc
I20260812 08:02:55.227458 26167 raft_consensus.cc:2749] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:02:55.228721 26012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "83539aa895e1497ab4b0a263cd4797cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e4ae4d92be471383bd23690e34b55b" is_pre_election: true
I20260812 08:02:55.228895 26012 raft_consensus.cc:2393] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83539aa895e1497ab4b0a263cd4797cc in current term 1: Already voted for candidate 164745ab533a4fd48893689e86e7c197 in this term.
I20260812 08:02:55.229302 26160 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: TxnStatusTablet state changed. Reason: New leader 164745ab533a4fd48893689e86e7c197. Latest consensus state: current_term: 1 leader_uuid: "164745ab533a4fd48893689e86e7c197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:55.229406 26160 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: This TxnStatusTablet replica's current role is: LEADER
I20260812 08:02:55.229342 25486 leader_election.cc:304] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83539aa895e1497ab4b0a263cd4797cc; no voters: 01e4ae4d92be471383bd23690e34b55b, 164745ab533a4fd48893689e86e7c197
I20260812 08:02:55.229589 26161 raft_consensus.cc:2749] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:02:55.229880 25348 catalog_manager.cc:5719] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 164745ab533a4fd48893689e86e7c197 (127.22.155.2). New cstate: current_term: 1 leader_uuid: "164745ab533a4fd48893689e86e7c197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:55.230566 26179 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260812 08:02:55.230795 26179 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260812 08:02:56.277572 26124 tablet.cc:2406] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:02:56.279131 25909 tablet.cc:2406] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:56.282097 26160 consensus_queue.cc:1048] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:56.285256 26184 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 164745ab533a4fd48893689e86e7c197)
I20260812 08:02:56.285372 26184 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:56.285558 26167 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: TxnStatusTablet state changed. Reason: New leader 164745ab533a4fd48893689e86e7c197. Latest consensus state: current_term: 1 leader_uuid: "164745ab533a4fd48893689e86e7c197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:56.285670 26167 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 08:02:56.286053 25822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "01e4ae4d92be471383bd23690e34b55b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "164745ab533a4fd48893689e86e7c197" is_pre_election: true
I20260812 08:02:56.286623 26184 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 164745ab533a4fd48893689e86e7c197 (127.22.155.2:37515), 83539aa895e1497ab4b0a263cd4797cc (127.22.155.1:44059)
I20260812 08:02:56.291144 26160 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "164745ab533a4fd48893689e86e7c197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:56.291256 26160 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197: This TxnStatusTablet replica's current role is: LEADER
I20260812 08:02:56.293872 26184 tablet_replica.cc:442] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "164745ab533a4fd48893689e86e7c197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } } }
I20260812 08:02:56.293996 26184 tablet_replica.cc:445] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 08:02:56.359735 25687 tablet.cc:2406] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:56.360253 26202 raft_consensus.cc:493] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:56.360428 26202 raft_consensus.cc:515] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "164745ab533a4fd48893689e86e7c197" member_type: VOTER last_known_addr { host: "127.22.155.2" port: 37515 } } peers { permanent_uuid: "01e4ae4d92be471383bd23690e34b55b" member_type: VOTER last_known_addr { host: "127.22.155.3" port: 44441 } } peers { permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 } }
I20260812 08:02:56.360846 25600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "01e4ae4d92be471383bd23690e34b55b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83539aa895e1497ab4b0a263cd4797cc" is_pre_election: true
I20260812 08:02:56.361084 26202 leader_election.cc:290] T 69291db43d384d4c96bdafe659f125ef P 83539aa895e1497ab4b0a263cd4797cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 164745ab533a4fd48893689e86e7c197 (127.22.155.2:37515), 01e4ae4d92be471383bd23690e34b55b (127.22.155.3:44441)
I20260812 08:02:56.361832 26160 consensus_queue.cc:1048] T 69291db43d384d4c96bdafe659f125ef P 164745ab533a4fd48893689e86e7c197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83539aa895e1497ab4b0a263cd4797cc" member_type: VOTER last_known_addr { host: "127.22.155.1" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:56.366322 25822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "83539aa895e1497ab4b0a263cd4797cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164745ab533a4fd48893689e86e7c197" is_pre_election: true
I20260812 08:02:56.366986 25936 leader_election.cc:304] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e4ae4d92be471383bd23690e34b55b; no voters: 164745ab533a4fd48893689e86e7c197, 83539aa895e1497ab4b0a263cd4797cc
I20260812 08:02:56.367378 26184 raft_consensus.cc:2749] T 69291db43d384d4c96bdafe659f125ef P 01e4ae4d92be471383bd23690e34b55b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 08:02:56.367909 26203 mvcc.cc:204] Tried to move back new op lower bound from 7317593195953401856 to 7317593191331889152. Current Snapshot: MvccSnapshot[applied={T|T < 7317593195707756544}]
I20260812 08:02:56.368463 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 25455
I20260812 08:02:56.379356 26012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69291db43d384d4c96bdafe659f125ef" candidate_uuid: "83539aa895e1497ab4b0a263cd4797cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e4ae4d92be471383bd23690e34b55b" is_pre_election: true
W20260812 08:02:56.387001 25938 connection.cc:573] server connection from 127.22.155.1:37685 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:56.387200 25721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.155.1:44059 (error 108)
I20260812 08:02:56.387897 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 25697
I20260812 08:02:56.404289 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 25910
I20260812 08:02:56.420642 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 25301
2026-08-12T08:02:56Z chronyd exiting
[       OK ] ClientFailoverOnNegotiationTimeoutITest.TestTxnSystemClientRetryOnPause (2116 ms)
[ RUN      ] ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster
2026-08-12T08:02:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:56Z Disabled control of system clock
I20260812 08:02:56.459782 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:43215
--webserver_interface=127.22.155.62
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
W20260812 08:02:56.564389 26220 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:56.564618 26220 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:56.564659 26220 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:56.567979 26220 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:56.568051 26220 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:56.568073 26220 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:56.568096 26220 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:56.568114 26220 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:56.571959 26220 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.155.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:56.573067 26220 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:56.574121 26220 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:56.579607 26229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.579608 26235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.579655 26233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:56.579772 26220 server_base.cc:1061] running on GCE node
I20260812 08:02:56.580377 26220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:56.581277 26220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:56.582465 26220 hybrid_clock.cc:648] HybridClock initialized: now 1786521776582451 us; error 38 us; skew 500 ppm
I20260812 08:02:56.584266 26220 webserver.cc:533] Webserver started at http://127.22.155.62:36619/ using document root <none> and password file <none>
I20260812 08:02:56.584864 26220 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:56.584972 26220 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:56.585202 26220 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:56.586781 26220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data/instance:
uuid: "de5ffd89b6244b84aaa06fd604842adb"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.587275 26220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal/instance:
uuid: "de5ffd89b6244b84aaa06fd604842adb"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.590495 26220 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:02:56.592546 26246 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:56.593518 26220 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:56.593663 26220 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
uuid: "de5ffd89b6244b84aaa06fd604842adb"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.593772 26220 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:56.611928 26220 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:56.612622 26220 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:56.612844 26220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:56.620107 26220 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.62:43215
I20260812 08:02:56.620143 26334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.62:43215 every 8 connection(s)
I20260812 08:02:56.621096 26220 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:56.623921 26337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:56.625064 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 26220
I20260812 08:02:56.625190 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-0/wal/instance
I20260812 08:02:56.625842 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.61:46525
--webserver_interface=127.22.155.61
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
I20260812 08:02:56.627820 26337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.638422 26337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:56.639304 26250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.155.61:46525: connect: Connection refused (error 111)
W20260812 08:02:56.640288 26337 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:46525: Network error: Client connection negotiation failed: client connection to 127.22.155.61:46525: connect: Connection refused (error 111)
I20260812 08:02:56.689548 26337 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } attempt: 1
W20260812 08:02:56.691179 26337 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:46525: Network error: Client connection negotiation failed: client connection to 127.22.155.61:46525: connect: Connection refused (error 111)
W20260812 08:02:56.735193 26340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:56.735448 26340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:56.735489 26340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:56.738741 26340 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:56.738804 26340 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:56.738824 26340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:56.738844 26340 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:56.738862 26340 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:56.743070 26340 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.61:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.155.61
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:56.744074 26340 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:56.745180 26340 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:56.750262 26348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.750276 26349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.750303 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:56.750610 26340 server_base.cc:1061] running on GCE node
I20260812 08:02:56.751091 26340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:56.751959 26340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:56.753135 26340 hybrid_clock.cc:648] HybridClock initialized: now 1786521776753121 us; error 29 us; skew 500 ppm
I20260812 08:02:56.754825 26340 webserver.cc:533] Webserver started at http://127.22.155.61:41353/ using document root <none> and password file <none>
I20260812 08:02:56.755380 26340 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:56.755477 26340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:56.755697 26340 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:56.757266 26340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data/instance:
uuid: "043297cabf284f648ff50da00636b24b"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.757737 26340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal/instance:
uuid: "043297cabf284f648ff50da00636b24b"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.759352 26337 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } attempt: 2
W20260812 08:02:56.760926 26337 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:46525: Network error: Client connection negotiation failed: client connection to 127.22.155.61:46525: connect: Connection refused (error 111)
I20260812 08:02:56.761418 26340 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:02:56.763387 26365 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:56.764375 26340 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:56.764515 26340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
uuid: "043297cabf284f648ff50da00636b24b"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.764619 26340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:56.780289 26340 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:56.781075 26340 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:56.781279 26340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:56.788733 26340 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.61:46525
I20260812 08:02:56.788753 26461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.61:46525 every 8 connection(s)
I20260812 08:02:56.789796 26340 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:56.792580 26462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:56.792793 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 26340
I20260812 08:02:56.792930 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-1/wal/instance
I20260812 08:02:56.793614 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.60:41135
--webserver_interface=127.22.155.60
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
I20260812 08:02:56.796057 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.804757 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.811493 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:56.812252 26366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
W20260812 08:02:56.813179 26462 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:41135: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
I20260812 08:02:56.852157 26337 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } attempt: 3
I20260812 08:02:56.856933 26337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:56.858418 26337 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:41135: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
I20260812 08:02:56.862445 26462 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } attempt: 1
W20260812 08:02:56.863873 26462 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:41135: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
I20260812 08:02:56.889595 26337 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } attempt: 1
W20260812 08:02:56.891185 26337 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:41135: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
W20260812 08:02:56.905637 26465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:56.905854 26465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:56.905894 26465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:56.909246 26465 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:56.909312 26465 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:56.909333 26465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:56.909354 26465 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:56.909422 26465 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:56.913333 26465 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.60:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.155.60
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:56.914398 26465 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:56.915515 26465 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:56.920760 26475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.920858 26479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:56.920797 26476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:56.921140 26465 server_base.cc:1061] running on GCE node
I20260812 08:02:56.921590 26465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:56.922501 26465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:56.923715 26465 hybrid_clock.cc:648] HybridClock initialized: now 1786521776923687 us; error 45 us; skew 500 ppm
I20260812 08:02:56.925523 26465 webserver.cc:533] Webserver started at http://127.22.155.60:45515/ using document root <none> and password file <none>
I20260812 08:02:56.926079 26465 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:56.926163 26465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:56.926379 26465 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:56.927894 26465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data/instance:
uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.928397 26465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal/instance:
uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.931614 26465 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20260812 08:02:56.932088 26462 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } attempt: 2
W20260812 08:02:56.933782 26462 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:41135: Network error: Client connection negotiation failed: client connection to 127.22.155.60:41135: connect: Connection refused (error 111)
I20260812 08:02:56.934012 26489 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:56.934986 26465 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:56.935087 26465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
format_stamp: "Formatted at 2026-08-12 08:02:56 on dist-test-slave-ffrd"
I20260812 08:02:56.935159 26465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:56.951819 26465 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:56.952443 26465 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:56.952594 26465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:56.960217 26573 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.60:41135 every 8 connection(s)
I20260812 08:02:56.960223 26465 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.60:41135
I20260812 08:02:56.961356 26465 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:56.964136 26578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:56.966413 26337 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } attempt: 2
I20260812 08:02:56.967712 26578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.969949 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 26465
I20260812 08:02:56.970068 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/master-2/wal/instance
I20260812 08:02:56.975962 26578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.978361 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: Bootstrap starting.
I20260812 08:02:56.980293 26578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:56.981848 26337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:56.982987 26337 log.cc:826] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:56.985646 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: No bootstrap required, opened a new log
I20260812 08:02:56.988608 26337 raft_consensus.cc:359] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:56.988792 26337 raft_consensus.cc:385] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:56.988853 26337 raft_consensus.cc:740] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5ffd89b6244b84aaa06fd604842adb, State: Initialized, Role: FOLLOWER
I20260812 08:02:56.989231 26578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: Bootstrap starting.
I20260812 08:02:56.989260 26337 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:56.989789 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:56.989938 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:56.990469 26337 sys_catalog.cc:565] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:56.991442 26578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:56.992321 26578 log.cc:826] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:56.994156 26578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: No bootstrap required, opened a new log
W20260812 08:02:56.995519 26608 catalog_manager.cc:1594] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:56.995594 26608 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:56.996101 26220 master_runner.cc:187] Error getting master registration for 127.22.155.61:46525: OK, instance_id { permanent_uuid: "043297cabf284f648ff50da00636b24b" instance_seqno: 1786521776786921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:56.998554 26220 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:56.998735 26578 raft_consensus.cc:359] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:56.999269 26578 raft_consensus.cc:385] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:56.999343 26592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:56.999395 26578 raft_consensus.cc:740] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165dc0007f6f48bfa5eb66492cd7f9aa, State: Initialized, Role: FOLLOWER
I20260812 08:02:56.999513 26592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.000047 26578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.000512 26592 leader_election.cc:290] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 043297cabf284f648ff50da00636b24b (127.22.155.61:46525), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:57.001410 26578 sys_catalog.cc:565] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:57.002082 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:57.002367 26533 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5ffd89b6244b84aaa06fd604842adb in term 0.
I20260812 08:02:57.002936 26247 leader_election.cc:304] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:57.003258 26592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:57.003357 26592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:57.003424 26592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:57.004516 26592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.004901 26592 leader_election.cc:290] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 1 election: Requested vote from peers 043297cabf284f648ff50da00636b24b (127.22.155.61:46525), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:57.005503 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.005673 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.008296 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
W20260812 08:02:57.008314 26632 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:57.008419 26632 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:57.008451 26533 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:57.009737 26533 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5ffd89b6244b84aaa06fd604842adb in term 1.
I20260812 08:02:57.010540 26247 leader_election.cc:304] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:57.010720 26592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:57.010927 26592 raft_consensus.cc:697] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 LEADER]: Becoming Leader. State: Replica: de5ffd89b6244b84aaa06fd604842adb, State: Running, Role: LEADER
I20260812 08:02:57.011242 26592 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.010506 26465 master_runner.cc:187] Error getting master registration for 127.22.155.61:46525: OK, instance_id { permanent_uuid: "043297cabf284f648ff50da00636b24b" instance_seqno: 1786521776786921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:57.012898 26465 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:57.013130 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5ffd89b6244b84aaa06fd604842adb. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.013253 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:57.013727 26638 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:57.014904 26638 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:57.024981 26462 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } attempt: 3
I20260812 08:02:57.032917 26462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: Bootstrap starting.
I20260812 08:02:57.035725 26462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:57.036885 26462 log.cc:826] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:57.041448 26533 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Refusing update from remote peer de5ffd89b6244b84aaa06fd604842adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:57.042173 26634 consensus_queue.cc:1048] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:57.046654 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5ffd89b6244b84aaa06fd604842adb. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.046785 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.050562 26462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: No bootstrap required, opened a new log
I20260812 08:02:57.052325 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.052475 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:57.052452 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.052543 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.053289 26643 mvcc.cc:204] Tried to move back new op lower bound from 7317593198754926592 to 7317593198641467392. Current Snapshot: MvccSnapshot[applied={T|T < 7317593198754926592}]
I20260812 08:02:57.054129 26462 raft_consensus.cc:359] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.054311 26462 raft_consensus.cc:385] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:57.054369 26462 raft_consensus.cc:740] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043297cabf284f648ff50da00636b24b, State: Initialized, Role: FOLLOWER
I20260812 08:02:57.054867 26462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.056066 26462 sys_catalog.cc:565] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:57.056114 26413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b"
I20260812 08:02:57.056303 26410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:57.056581 26415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:57.058785 26592 consensus_queue.cc:1048] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:57.062855 26638 catalog_manager.cc:1383] Generated new cluster ID: 2959a04f057e430bb9565f86e18a8744
I20260812 08:02:57.062930 26638 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:57.064893 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.065105 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.065253 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5ffd89b6244b84aaa06fd604842adb. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.065388 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.065531 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.065652 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.070010 26676 mvcc.cc:204] Tried to move back new op lower bound from 7317593198754926592 to 7317593198641467392. Current Snapshot: MvccSnapshot[applied={T|T < 7317593198754926592}]
I20260812 08:02:57.070811 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de5ffd89b6244b84aaa06fd604842adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.070926 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:57.073640 26667 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:57.073721 26667 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:57.076699 26638 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:57.077905 26638 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:57.093493 26638 catalog_manager.cc:6092] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: Generated new TSK 0
I20260812 08:02:57.094290 26638 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:57.111577 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.1:0
--local_ip_for_outbound_sockets=127.22.155.1
--webserver_interface=127.22.155.1
--webserver_port=0
--tserver_master_addrs=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--builtin_ntp_servers=127.22.155.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
W20260812 08:02:57.286711 26683 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:57.287000 26683 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:57.287088 26683 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:57.292068 26683 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:57.292173 26683 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:57.292243 26683 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:02:57.292315 26683 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.1
I20260812 08:02:57.298316 26683 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.155.1
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:43215,127.22.155.61:46525,127.22.155.60:41135
--never_fsync=true
--heap_profile_path=/tmp/kudu.26683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:57.299813 26683 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:57.301307 26683 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:57.305033 26683 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:02:57.309597 26691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:57.309669 26689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:57.309592 26693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:57.309832 26683 server_base.cc:1061] running on GCE node
I20260812 08:02:57.310302 26683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:57.310941 26683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:57.312150 26683 hybrid_clock.cc:648] HybridClock initialized: now 1786521777312125 us; error 33 us; skew 500 ppm
I20260812 08:02:57.314301 26683 webserver.cc:533] Webserver started at http://127.22.155.1:35931/ using document root <none> and password file <none>
I20260812 08:02:57.315032 26683 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:57.315124 26683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:57.315356 26683 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:57.317448 26683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/instance:
uuid: "a239a0261565480d80fc0be46ca96e3b"
format_stamp: "Formatted at 2026-08-12 08:02:57 on dist-test-slave-ffrd"
I20260812 08:02:57.318064 26683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance:
uuid: "a239a0261565480d80fc0be46ca96e3b"
format_stamp: "Formatted at 2026-08-12 08:02:57 on dist-test-slave-ffrd"
I20260812 08:02:57.322422 26683 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:02:57.324635 26703 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:57.325719 26683 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:02:57.325868 26683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
uuid: "a239a0261565480d80fc0be46ca96e3b"
format_stamp: "Formatted at 2026-08-12 08:02:57 on dist-test-slave-ffrd"
I20260812 08:02:57.325975 26683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:57.340627 26683 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:57.341414 26683 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:57.341622 26683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:57.343288 26683 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:57.343364 26683 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:57.343437 26683 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:57.343477 26683 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:57.354975 26683 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.1:40429
I20260812 08:02:57.355074 26884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.1:40429 every 8 connection(s)
I20260812 08:02:57.356349 26683 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:57.358775 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 26683
I20260812 08:02:57.358884 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance
I20260812 08:02:57.363658 26896 heartbeater.cc:344] Connected to a master server at 127.22.155.60:41135
I20260812 08:02:57.363862 26896 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:57.364483 26896 heartbeater.cc:507] Master 127.22.155.60:41135 requested a full tablet report, sending...
I20260812 08:02:57.365762 26897 heartbeater.cc:344] Connected to a master server at 127.22.155.62:43215
I20260812 08:02:57.365867 26897 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:57.366053 26897 heartbeater.cc:507] Master 127.22.155.62:43215 requested a full tablet report, sending...
I20260812 08:02:57.366302 26901 heartbeater.cc:344] Connected to a master server at 127.22.155.61:46525
I20260812 08:02:57.366384 26901 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:57.366557 26901 heartbeater.cc:507] Master 127.22.155.61:46525 requested a full tablet report, sending...
I20260812 08:02:57.367167 26271 ts_manager.cc:194] Registered new tserver with Master: a239a0261565480d80fc0be46ca96e3b (127.22.155.1:40429)
I20260812 08:02:57.368798 26271 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.1:56075
I20260812 08:02:57.369860 26397 ts_manager.cc:194] Registered new tserver with Master: a239a0261565480d80fc0be46ca96e3b (127.22.155.1:40429)
I20260812 08:02:57.369860 26515 ts_manager.cc:194] Registered new tserver with Master: a239a0261565480d80fc0be46ca96e3b (127.22.155.1:40429)
I20260812 08:02:57.392094 23148 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 08:02:57.398466 26897 heartbeater.cc:499] Master 127.22.155.62:43215 was elected leader, sending a full tablet report...
I20260812 08:02:57.465803 26672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de5ffd89b6244b84aaa06fd604842adb)
I20260812 08:02:57.465999 26672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.466990 26672 leader_election.cc:290] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:57.467592 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:57.471918 26614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de5ffd89b6244b84aaa06fd604842adb)
I20260812 08:02:57.472031 26614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.472900 26614 leader_election.cc:290] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 043297cabf284f648ff50da00636b24b (127.22.155.61:46525)
I20260812 08:02:57.473336 26410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:57.473496 26410 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 1.
I20260812 08:02:57.473870 26497 leader_election.cc:304] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, 165dc0007f6f48bfa5eb66492cd7f9aa; no voters: 
I20260812 08:02:57.474129 26614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:57.474205 26614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader de5ffd89b6244b84aaa06fd604842adb)
I20260812 08:02:57.474261 26614 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:57.475158 26614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.475503 26614 leader_election.cc:290] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 election: Requested vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 043297cabf284f648ff50da00636b24b (127.22.155.61:46525)
I20260812 08:02:57.475857 26410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b"
I20260812 08:02:57.475968 26410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:57.477056 26410 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 2.
I20260812 08:02:57.477376 26497 leader_election.cc:304] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, 165dc0007f6f48bfa5eb66492cd7f9aa; no voters: 
I20260812 08:02:57.477620 26614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260812 08:02:57.477842 26614 raft_consensus.cc:697] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 LEADER]: Becoming Leader. State: Replica: 165dc0007f6f48bfa5eb66492cd7f9aa, State: Running, Role: LEADER
I20260812 08:02:57.478121 26614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.479239 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.479326 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:57.479640 26924 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:57.480787 26924 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:57.481348 26924 catalog_manager.cc:1295] Loaded cluster ID: 2959a04f057e430bb9565f86e18a8744
I20260812 08:02:57.481421 26924 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:57.483461 26924 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:57.484210 26924 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: Loaded TSK: 0
I20260812 08:02:57.485323 26924 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:57.508009 26410 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Refusing update from remote peer 165dc0007f6f48bfa5eb66492cd7f9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 08:02:57.508509 26923 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 08:02:57.509619 26672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.509732 26672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.510560 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.510675 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:57.511111 26672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.511217 26672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.529421 26896 heartbeater.cc:499] Master 127.22.155.60:41135 was elected leader, sending a full tablet report...
W20260812 08:02:57.565814 26492 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.155.62:43215 timed out after 0.092s (SENT)
W20260812 08:02:57.659498 26492 leader_election.cc:336] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215): Timed out: RequestConsensusVote RPC to 127.22.155.62:43215 timed out after 0.092s (SENT)
W20260812 08:02:57.676803 26366 leader_election.cc:336] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215): Timed out: RequestConsensusVote RPC to 127.22.155.62:43215 timed out after 0.104s (SENT)
I20260812 08:02:57.677018 26366 leader_election.cc:304] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 043297cabf284f648ff50da00636b24b; no voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb
I20260812 08:02:57.677311 26672 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 08:02:57.688993 26492 leader_election.cc:336] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215): Timed out: RequestConsensusVote RPC to 127.22.155.62:43215 timed out after 0.106s (SENT)
W20260812 08:02:57.953382 26922 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:57.421742 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.62:43215 (local address 127.0.0.1:58854)
0812 08:02:57.421889 (+   147us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:57.421903 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:57.421969 (+    66us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:57.953202 (+531233us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.62:43215: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 08:02:57.954120 26911 master_proxy_rpc.cc:203] Re-attempting ListTables request to leader Master (127.22.155.62:43215)
I20260812 08:02:57.966873 26279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:57.967286 26280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:57.967502 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb"
I20260812 08:02:57.967693 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:57.968101 26286 raft_consensus.cc:3055] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 LEADER]: Stepping down as leader of term 1
I20260812 08:02:57.968169 26286 raft_consensus.cc:740] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de5ffd89b6244b84aaa06fd604842adb, State: Running, Role: LEADER
I20260812 08:02:57.968336 26286 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:57.968576 26286 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:57.969789 26286 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Refusing update from remote peer 165dc0007f6f48bfa5eb66492cd7f9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 08:02:57.970319 26923 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 08:02:57.970582 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:57.970731 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 2.
I20260812 08:02:57.970948 26287 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb"
I20260812 08:02:57.972010 26287 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 2.
I20260812 08:02:57.973250 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.973371 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:57.973448 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:57.973501 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
W20260812 08:02:57.975119 26939 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:57.969789 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.22.155.62:43215)
0812 08:02:57.971530 (+  1741us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:57.971534 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:57.971548 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:57.971588 (+    40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:57.971592 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:57.971641 (+    49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:57.971774 (+   133us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58854: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":82,"threads_started":1}
I20260812 08:02:57.996830 26608 catalog_manager.cc:1638] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:58.047101 26672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 165dc0007f6f48bfa5eb66492cd7f9aa)
I20260812 08:02:58.047241 26672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.047741 26672 leader_election.cc:290] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.047827 26592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 165dc0007f6f48bfa5eb66492cd7f9aa)
I20260812 08:02:58.047968 26592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.048270 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:58.048408 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 043297cabf284f648ff50da00636b24b in term 2.
I20260812 08:02:58.048450 26592 leader_election.cc:290] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 043297cabf284f648ff50da00636b24b (127.22.155.61:46525), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.048834 26366 leader_election.cc:304] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.048837 26410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:58.049017 26410 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5ffd89b6244b84aaa06fd604842adb in term 2.
I20260812 08:02:58.049115 26672 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:58.049203 26672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 165dc0007f6f48bfa5eb66492cd7f9aa)
I20260812 08:02:58.049256 26672 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:58.049333 26250 leader_election.cc:304] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.049548 26592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:58.049645 26592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 165dc0007f6f48bfa5eb66492cd7f9aa)
I20260812 08:02:58.049706 26592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:58.050292 26672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.050673 26672 leader_election.cc:290] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 election: Requested vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.050823 26592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.050980 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb"
I20260812 08:02:58.051115 26288 raft_consensus.cc:2393] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 043297cabf284f648ff50da00636b24b in current term 3: Already voted for candidate de5ffd89b6244b84aaa06fd604842adb in this term.
I20260812 08:02:58.051210 26592 leader_election.cc:290] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 election: Requested vote from peers 043297cabf284f648ff50da00636b24b (127.22.155.61:46525), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.051638 26410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b"
I20260812 08:02:58.051802 26410 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate de5ffd89b6244b84aaa06fd604842adb in current term 3: Already voted for candidate 043297cabf284f648ff50da00636b24b in this term.
I20260812 08:02:58.074904 26667 catalog_manager.cc:1295] Loaded cluster ID: 2959a04f057e430bb9565f86e18a8744
I20260812 08:02:58.075107 26667 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: loading cluster ID for follower catalog manager: success
I20260812 08:02:58.076840 26667 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: acquiring CA information for follower catalog manager: success
I20260812 08:02:58.077911 26667 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 08:02:58.214135 26366 leader_election.cc:336] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135): Timed out: RequestConsensusVote RPC to 127.22.155.60:41135 timed out after 0.081s (SENT)
I20260812 08:02:58.214282 26366 leader_election.cc:304] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 043297cabf284f648ff50da00636b24b; no voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb
I20260812 08:02:58.214527 26672 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260812 08:02:58.243690 26366 leader_election.cc:336] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135): Timed out: RequestConsensusVote RPC to 127.22.155.60:41135 timed out after 0.097s (SENT)
W20260812 08:02:58.249269 26247 leader_election.cc:336] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135): Timed out: RequestConsensusVote RPC to 127.22.155.60:41135 timed out after 0.099s (SENT)
W20260812 08:02:58.272863 26247 leader_election.cc:336] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135): Timed out: RequestConsensusVote RPC to 127.22.155.60:41135 timed out after 0.110s (SENT)
I20260812 08:02:58.273066 26247 leader_election.cc:304] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de5ffd89b6244b84aaa06fd604842adb; no voters: 043297cabf284f648ff50da00636b24b, 165dc0007f6f48bfa5eb66492cd7f9aa
I20260812 08:02:58.273314 26592 raft_consensus.cc:2749] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 08:02:58.316220 26672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:58.316352 26672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.316816 26672 leader_election.cc:290] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.317224 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:58.317344 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 043297cabf284f648ff50da00636b24b in term 3.
I20260812 08:02:58.317649 26366 leader_election.cc:304] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.317858 26672 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:02:58.317945 26672 raft_consensus.cc:493] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:58.317996 26672 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:58.319749 26672 raft_consensus.cc:515] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.320091 26672 leader_election.cc:290] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 4 election: Requested vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.320426 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb"
I20260812 08:02:58.320526 26288 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:02:58.321367 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 043297cabf284f648ff50da00636b24b in term 4.
I20260812 08:02:58.321663 26366 leader_election.cc:304] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043297cabf284f648ff50da00636b24b, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.321864 26672 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 FOLLOWER]: Leader election won for term 4
I20260812 08:02:58.322041 26672 raft_consensus.cc:697] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 LEADER]: Becoming Leader. State: Replica: 043297cabf284f648ff50da00636b24b, State: Running, Role: LEADER
I20260812 08:02:58.322382 26672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.323535 26947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043297cabf284f648ff50da00636b24b. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.323635 26947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:58.323936 26948 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:58.324815 26948 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:58.325233 26948 catalog_manager.cc:1295] Loaded cluster ID: 2959a04f057e430bb9565f86e18a8744
I20260812 08:02:58.325281 26948 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:58.325800 26948 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:58.326190 26948 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b: Loaded TSK: 0
I20260812 08:02:58.326773 26948 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:58.351027 26288 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Refusing update from remote peer 043297cabf284f648ff50da00636b24b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260812 08:02:58.351588 26947 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 08:02:58.352686 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043297cabf284f648ff50da00636b24b. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.352849 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.353814 26672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.353933 26672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:58.354157 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.354271 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.375783 26901 heartbeater.cc:499] Master 127.22.155.61:46525 was elected leader, sending a full tablet report...
W20260812 08:02:58.529309 26928 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:57.999191 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:41135 (local address 127.0.0.1:41046)
0812 08:02:57.999374 (+   183us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:57.999390 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:57.999474 (+    84us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:58.529204 (+529730us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.60:41135: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 08:02:58.541074 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.541421 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.541636 26533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.541886 26533 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 LEADER]: Stepping down as leader of term 2
I20260812 08:02:58.541935 26533 raft_consensus.cc:740] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 165dc0007f6f48bfa5eb66492cd7f9aa, State: Running, Role: LEADER
I20260812 08:02:58.542050 26516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.542285 26518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.542531 26519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.542672 26288 raft_consensus.cc:1240] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Rejecting Update request from peer 165dc0007f6f48bfa5eb66492cd7f9aa for earlier term 2. Current term is 4. Ops: []
I20260812 08:02:58.542768 26524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.542986 26526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.543172 26614 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 08:02:58.543231 26527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.543442 26528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.543682 26529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.543931 26533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.544039 26521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "043297cabf284f648ff50da00636b24b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa"
I20260812 08:02:58.544227 26533 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 2 FOLLOWER]: Advancing to term 4
I20260812 08:02:58.544574 26410 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 LEADER]: Rejecting Update request from peer 165dc0007f6f48bfa5eb66492cd7f9aa for earlier term 2. Current term is 4. Ops: []
W20260812 08:02:58.545238 26962 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:58.543222 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.22.155.60:41135)
0812 08:02:58.543552 (+   330us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:58.543556 (+     4us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:58.543570 (+    14us) server_negotiation.cc:408] Connection header received
0812 08:02:58.543609 (+    39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:58.543612 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:58.543668 (+    56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:58.543800 (+   132us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41046: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":79,"threads_started":1}
I20260812 08:02:58.545571 26533 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Refusing update from remote peer 043297cabf284f648ff50da00636b24b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260812 08:02:58.545681 26614 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260812 08:02:58.546252 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.546360 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.546860 26672 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 08:02:58.548359 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043297cabf284f648ff50da00636b24b. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.548347 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "043297cabf284f648ff50da00636b24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.548470 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.548478 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.609575 26592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 043297cabf284f648ff50da00636b24b)
I20260812 08:02:58.609723 26592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.610240 26592 leader_election.cc:290] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 043297cabf284f648ff50da00636b24b (127.22.155.61:46525), 165dc0007f6f48bfa5eb66492cd7f9aa (127.22.155.60:41135)
I20260812 08:02:58.610683 26521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" is_pre_election: true
I20260812 08:02:58.623978 26614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 043297cabf284f648ff50da00636b24b)
I20260812 08:02:58.624130 26614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.624670 26614 leader_election.cc:290] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 043297cabf284f648ff50da00636b24b (127.22.155.61:46525)
I20260812 08:02:58.625255 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb" is_pre_election: true
I20260812 08:02:58.625448 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 4.
I20260812 08:02:58.625828 26492 leader_election.cc:304] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.626037 26614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:02:58.626121 26614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 043297cabf284f648ff50da00636b24b)
I20260812 08:02:58.626168 26614 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:58.627211 26614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.627570 26614 leader_election.cc:290] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 election: Requested vote from peers de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215), 043297cabf284f648ff50da00636b24b (127.22.155.61:46525)
I20260812 08:02:58.627975 26288 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "de5ffd89b6244b84aaa06fd604842adb"
I20260812 08:02:58.628088 26288 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:58.628846 26288 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 5.
I20260812 08:02:58.629192 26492 leader_election.cc:304] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165dc0007f6f48bfa5eb66492cd7f9aa, de5ffd89b6244b84aaa06fd604842adb; no voters: 
I20260812 08:02:58.629418 26614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 5 FOLLOWER]: Leader election won for term 5
I20260812 08:02:58.629527 26614 raft_consensus.cc:697] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [term 5 LEADER]: Becoming Leader. State: Replica: 165dc0007f6f48bfa5eb66492cd7f9aa, State: Running, Role: LEADER
I20260812 08:02:58.629671 26614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:58.630336 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.630429 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:58.630748 26975 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:58.631639 26975 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:58.632036 26975 catalog_manager.cc:1295] Loaded cluster ID: 2959a04f057e430bb9565f86e18a8744
I20260812 08:02:58.632083 26975 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:58.632615 26975 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:58.633091 26975 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa: Loaded TSK: 0
I20260812 08:02:58.633401 26975 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:58.659299 26288 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 5 FOLLOWER]: Refusing update from remote peer 165dc0007f6f48bfa5eb66492cd7f9aa: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 08:02:58.659835 26923 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 08:02:58.660916 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.661118 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.661929 26614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.662050 26614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:58.662650 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:58.662775 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:58.678583 26896 heartbeater.cc:499] Master 127.22.155.60:41135 was elected leader, sending a full tablet report...
W20260812 08:02:58.811841 26250 leader_election.cc:336] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 043297cabf284f648ff50da00636b24b (127.22.155.61:46525): Timed out: RequestConsensusVote RPC to 127.22.155.61:46525 timed out after 0.100s (SENT)
I20260812 08:02:58.812034 26250 leader_election.cc:304] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de5ffd89b6244b84aaa06fd604842adb; no voters: 043297cabf284f648ff50da00636b24b, 165dc0007f6f48bfa5eb66492cd7f9aa
I20260812 08:02:58.812276 26592 raft_consensus.cc:2749] T 00000000000000000000000000000000 P de5ffd89b6244b84aaa06fd604842adb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 08:02:58.838945 26497 leader_election.cc:336] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 043297cabf284f648ff50da00636b24b (127.22.155.61:46525): Timed out: RequestConsensusVote RPC to 127.22.155.61:46525 timed out after 0.105s (SENT)
W20260812 08:02:58.840325 26497 leader_election.cc:336] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 043297cabf284f648ff50da00636b24b (127.22.155.61:46525): Timed out: RequestConsensusVote RPC to 127.22.155.61:46525 timed out after 0.107s (SENT)
W20260812 08:02:59.073319 26955 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:58.565061 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.61:46525 (local address 127.0.0.1:54486)
0812 08:02:58.565205 (+   144us) negotiation.cc:107] Waiting for socket to connect
0812 08:02:58.565219 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:02:58.565298 (+    79us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:02:59.073210 (+507912us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.61:46525: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 08:02:59.073853 26915 master_proxy_rpc.cc:203] Re-attempting ListTables request to leader Master (127.22.155.61:46525)
I20260812 08:02:59.082803 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 26683
I20260812 08:02:59.083998 26399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:59.084434 26403 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de5ffd89b6244b84aaa06fd604842adb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:59.084712 26407 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b"
I20260812 08:02:59.084995 26409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:59.085253 26410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b" is_pre_election: true
I20260812 08:02:59.085582 26413 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 LEADER]: Stepping down as leader of term 4
I20260812 08:02:59.085656 26413 raft_consensus.cc:740] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 043297cabf284f648ff50da00636b24b, State: Running, Role: LEADER
I20260812 08:02:59.085850 26413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } }
I20260812 08:02:59.086191 26413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:02:59.087167 26413 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 5 FOLLOWER]: Refusing update from remote peer 165dc0007f6f48bfa5eb66492cd7f9aa: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260812 08:02:59.087427 26415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "043297cabf284f648ff50da00636b24b"
I20260812 08:02:59.088495 26415 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165dc0007f6f48bfa5eb66492cd7f9aa in term 5.
W20260812 08:02:59.089514 26983 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:02:59.088323 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.22.155.61:46525)
0812 08:02:59.088599 (+   276us) server_negotiation.cc:207] Beginning negotiation
0812 08:02:59.088604 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:02:59.088620 (+    16us) server_negotiation.cc:408] Connection header received
0812 08:02:59.088663 (+    43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:02:59.088669 (+     6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:02:59.088731 (+    62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:02:59.088912 (+   181us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54486: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260812 08:02:59.093650 26614 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 08:02:59.095145 26947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165dc0007f6f48bfa5eb66492cd7f9aa. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:59.095289 26947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.095417 26947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:59.095513 26947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043297cabf284f648ff50da00636b24b [sys.catalog]: This master's current role is: FOLLOWER
W20260812 08:02:59.101497 26369 connection.cc:573] server connection from 127.22.155.1:58889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:02:59.102205 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 26220
W20260812 08:02:59.113375 26492 connection.cc:573] client connection to 127.22.155.62:43215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:02:59.113698 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 26340
W20260812 08:02:59.121953 26492 consensus_peers.cc:597] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa -> Peer de5ffd89b6244b84aaa06fd604842adb (127.22.155.62:43215): Couldn't send request to peer de5ffd89b6244b84aaa06fd604842adb. Status: Network error: Client connection negotiation failed: client connection to 127.22.155.62:43215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:59.124030 26961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:59.124197 26961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:59.128636 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5ffd89b6244b84aaa06fd604842adb" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 43215 } } peers { permanent_uuid: "043297cabf284f648ff50da00636b24b" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 46525 } } peers { permanent_uuid: "165dc0007f6f48bfa5eb66492cd7f9aa" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 41135 } } }
I20260812 08:02:59.128796 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165dc0007f6f48bfa5eb66492cd7f9aa [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:59.128876 26497 connection.cc:573] client connection to 127.22.155.61:46525 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:02:59.129174 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 26465
2026-08-12T08:02:59Z chronyd exiting
[       OK ] ClientFailoverOnNegotiationTimeoutITest.Kudu2021ConnectToMaster (2707 ms)
[ RUN      ] ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster
2026-08-12T08:02:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:59Z Disabled control of system clock
I20260812 08:02:59.168484 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:45729
--webserver_interface=127.22.155.62
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
W20260812 08:02:59.276582 26994 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:59.276849 26994 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:59.276898 26994 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:59.280524 26994 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:59.280614 26994 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:59.280634 26994 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:59.280654 26994 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:59.280673 26994 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:59.284860 26994 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.62:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.155.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.26994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:59.286088 26994 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:59.287269 26994 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:59.292775 27006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:59.292781 27002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:59.292781 27003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:59.293164 26994 server_base.cc:1061] running on GCE node
I20260812 08:02:59.293686 26994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:59.294593 26994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:59.295795 26994 hybrid_clock.cc:648] HybridClock initialized: now 1786521779295791 us; error 48 us; skew 500 ppm
I20260812 08:02:59.297610 26994 webserver.cc:533] Webserver started at http://127.22.155.62:42441/ using document root <none> and password file <none>
I20260812 08:02:59.298170 26994 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:59.298256 26994 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:59.298488 26994 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:59.299998 26994 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data/instance:
uuid: "6fcf06aba4f947109641fc459b340016"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.300490 26994 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal/instance:
uuid: "6fcf06aba4f947109641fc459b340016"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.303740 26994 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:02:59.305888 27022 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:59.306828 26994 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:59.306978 26994 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
uuid: "6fcf06aba4f947109641fc459b340016"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.307085 26994 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:59.337868 26994 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:59.338619 26994 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:59.338848 26994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:59.346416 27104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.62:45729 every 8 connection(s)
I20260812 08:02:59.346421 26994 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.62:45729
I20260812 08:02:59.347419 26994 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:59.350107 27105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:59.353360 27105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.354250 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 26994
I20260812 08:02:59.354390 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-0/wal/instance
I20260812 08:02:59.355080 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.61:42727
--webserver_interface=127.22.155.61
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
I20260812 08:02:59.363457 27105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:59.364213 27028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.155.61:42727: connect: Connection refused (error 111)
W20260812 08:02:59.365069 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:42727: Network error: Client connection negotiation failed: client connection to 127.22.155.61:42727: connect: Connection refused (error 111)
I20260812 08:02:59.414361 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } attempt: 1
W20260812 08:02:59.415932 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:42727: Network error: Client connection negotiation failed: client connection to 127.22.155.61:42727: connect: Connection refused (error 111)
W20260812 08:02:59.460340 27115 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:59.460628 27115 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:59.460672 27115 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:59.464529 27115 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:59.464592 27115 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:59.464612 27115 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:59.464666 27115 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:59.464705 27115 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:59.468503 27115 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.61:42727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.155.61
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.27115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:59.469643 27115 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:59.470727 27115 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:59.476147 27126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:59.476176 27130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:59.476356 27115 server_base.cc:1061] running on GCE node
W20260812 08:02:59.476171 27127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:59.477068 27115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:59.477933 27115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:59.479157 27115 hybrid_clock.cc:648] HybridClock initialized: now 1786521779479141 us; error 33 us; skew 500 ppm
I20260812 08:02:59.481181 27115 webserver.cc:533] Webserver started at http://127.22.155.61:41983/ using document root <none> and password file <none>
I20260812 08:02:59.481758 27115 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:59.481868 27115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:59.482067 27115 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:59.483623 27115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data/instance:
uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.484151 27115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal/instance:
uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.484130 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } attempt: 2
W20260812 08:02:59.485997 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.61:42727: Network error: Client connection negotiation failed: client connection to 127.22.155.61:42727: connect: Connection refused (error 111)
I20260812 08:02:59.488915 27115 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260812 08:02:59.491880 27137 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:59.492909 27115 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:59.493079 27115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.493161 27115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:59.511690 27115 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:59.512301 27115 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:59.512447 27115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:59.519706 27216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.61:42727 every 8 connection(s)
I20260812 08:02:59.519706 27115 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.61:42727
I20260812 08:02:59.520717 27115 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:59.521811 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 27115
I20260812 08:02:59.521930 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-1/wal/instance
I20260812 08:02:59.522688 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.60:35659
--webserver_interface=127.22.155.60
--webserver_port=0
--builtin_ntp_servers=127.22.155.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000 with env {}
I20260812 08:02:59.524286 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:59.527845 27217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.537242 27217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.543108 27217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:59.543859 27138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
W20260812 08:02:59.544759 27217 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
I20260812 08:02:59.577232 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } attempt: 3
I20260812 08:02:59.582248 27105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:59.583679 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
I20260812 08:02:59.594060 27217 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 1
W20260812 08:02:59.595686 27217 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
I20260812 08:02:59.614892 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 1
W20260812 08:02:59.616490 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
W20260812 08:02:59.635774 27220 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:59.636055 27220 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:59.636096 27220 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:59.639335 27220 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:02:59.639400 27220 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:59.639420 27220 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:59.639439 27220 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:59.639456 27220 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:59.643348 27220 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.155.60:35659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.155.60
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.27220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:59.644409 27220 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:59.645840 27220 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:59.651625 27229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:59.651808 27235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:59.651932 27233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:59.652150 27220 server_base.cc:1061] running on GCE node
I20260812 08:02:59.652827 27220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:59.654042 27220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:59.655253 27220 hybrid_clock.cc:648] HybridClock initialized: now 1786521779655224 us; error 36 us; skew 500 ppm
I20260812 08:02:59.657663 27220 webserver.cc:533] Webserver started at http://127.22.155.60:38835/ using document root <none> and password file <none>
I20260812 08:02:59.658423 27220 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:59.658502 27220 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:59.658735 27220 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:59.661381 27220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data/instance:
uuid: "a9eb51c31866473582eec6326b40f76c"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.662153 27220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal/instance:
uuid: "a9eb51c31866473582eec6326b40f76c"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.663918 27217 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 2
W20260812 08:02:59.665380 27217 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
I20260812 08:02:59.666833 27220 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20260812 08:02:59.669827 27242 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:59.671120 27220 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:02:59.671299 27220 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
uuid: "a9eb51c31866473582eec6326b40f76c"
format_stamp: "Formatted at 2026-08-12 08:02:59 on dist-test-slave-ffrd"
I20260812 08:02:59.671422 27220 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:59.688721 27220 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:59.689819 27220 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:59.690050 27220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:59.691684 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 2
W20260812 08:02:59.693457 27105 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.155.60:35659: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: connect: Connection refused (error 111)
I20260812 08:02:59.700845 27318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.60:35659 every 8 connection(s)
I20260812 08:02:59.700872 27220 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.60:35659
I20260812 08:02:59.702384 27220 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:59.705382 27319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:59.708853 27319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.709494 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 27220
I20260812 08:02:59.709626 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/master-2/wal/instance
I20260812 08:02:59.718766 27319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.723834 27319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:59.730520 27319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: Bootstrap starting.
I20260812 08:02:59.733271 27319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:59.734033 27319 log.cc:826] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:59.736027 27319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: No bootstrap required, opened a new log
I20260812 08:02:59.738927 27319 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.739082 27319 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:59.739116 27319 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9eb51c31866473582eec6326b40f76c, State: Initialized, Role: FOLLOWER
I20260812 08:02:59.739439 27319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.739821 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.739950 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.740444 27319 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:02:59.745235 27357 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:59.745308 27357 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:59.744011 27220 master_runner.cc:187] Error getting master registration for 127.22.155.62:45729: OK, instance_id { permanent_uuid: "6fcf06aba4f947109641fc459b340016" instance_seqno: 1786521779344372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:59.746131 27220 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:59.754664 27332 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:59.754855 27332 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.755918 27332 leader_election.cc:290] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fcf06aba4f947109641fc459b340016 (127.22.155.62:45729), bbe07b2b9c554d7ebfb03d50c76d3c2a (127.22.155.61:42727)
I20260812 08:02:59.756548 27217 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 3
I20260812 08:02:59.763213 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: Bootstrap starting.
I20260812 08:02:59.766150 27217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:59.767004 27217 log.cc:826] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:59.769007 27217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: No bootstrap required, opened a new log
I20260812 08:02:59.772871 27217 raft_consensus.cc:359] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.773082 27217 raft_consensus.cc:385] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:59.773150 27217 raft_consensus.cc:740] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe07b2b9c554d7ebfb03d50c76d3c2a, State: Initialized, Role: FOLLOWER
I20260812 08:02:59.773595 27217 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.774099 27364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.774811 27217 sys_catalog.cc:565] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:59.774883 27176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9eb51c31866473582eec6326b40f76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" is_pre_election: true
I20260812 08:02:59.775559 27176 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9eb51c31866473582eec6326b40f76c in term 0.
I20260812 08:02:59.776046 27248 leader_election.cc:304] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9eb51c31866473582eec6326b40f76c, bbe07b2b9c554d7ebfb03d50c76d3c2a; no voters: 
I20260812 08:02:59.776309 27332 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:59.774212 27364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.776391 27332 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:59.776448 27332 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:59.777961 27332 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.778394 27332 leader_election.cc:290] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [CANDIDATE]: Term 1 election: Requested vote from peers 6fcf06aba4f947109641fc459b340016 (127.22.155.62:45729), bbe07b2b9c554d7ebfb03d50c76d3c2a (127.22.155.61:42727)
I20260812 08:02:59.779997 27176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9eb51c31866473582eec6326b40f76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a"
I20260812 08:02:59.780125 27176 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:59.780994 27176 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9eb51c31866473582eec6326b40f76c in term 1.
I20260812 08:02:59.781335 27248 leader_election.cc:304] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9eb51c31866473582eec6326b40f76c, bbe07b2b9c554d7ebfb03d50c76d3c2a; no voters: 
I20260812 08:02:59.781514 27332 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:59.781716 27332 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 LEADER]: Becoming Leader. State: Replica: a9eb51c31866473582eec6326b40f76c, State: Running, Role: LEADER
I20260812 08:02:59.782021 27332 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
W20260812 08:02:59.782667 27379 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:59.782850 27379 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:59.783608 27115 master_runner.cc:187] Error getting master registration for 127.22.155.62:45729: OK, instance_id { permanent_uuid: "6fcf06aba4f947109641fc459b340016" instance_seqno: 1786521779344372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:59.785888 27115 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:59.787714 27383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9eb51c31866473582eec6326b40f76c. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.787810 27383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:59.788206 27385 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:59.789238 27385 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:59.792572 27105 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } attempt: 3
I20260812 08:02:59.797713 27176 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 1 FOLLOWER]: Refusing update from remote peer a9eb51c31866473582eec6326b40f76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:59.799037 27388 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:59.801424 27105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: Bootstrap starting.
I20260812 08:02:59.803160 27364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9eb51c31866473582eec6326b40f76c. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.803282 27364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.804225 27105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:59.805307 27105 log.cc:826] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:59.807478 27105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: No bootstrap required, opened a new log
I20260812 08:02:59.811292 27105 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.811488 27105 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:59.811549 27105 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcf06aba4f947109641fc459b340016, State: Initialized, Role: FOLLOWER
I20260812 08:02:59.812014 27105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:02:59.812711 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.813105 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.813627 27105 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:59.813702 27063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9eb51c31866473582eec6326b40f76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fcf06aba4f947109641fc459b340016"
I20260812 08:02:59.813946 27063 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:59.815137 27063 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9eb51c31866473582eec6326b40f76c in term 1.
I20260812 08:02:59.816226 27061 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Refusing update from remote peer a9eb51c31866473582eec6326b40f76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:59.816349 27067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9eb51c31866473582eec6326b40f76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fcf06aba4f947109641fc459b340016" is_pre_election: true
I20260812 08:02:59.817217 27067 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a9eb51c31866473582eec6326b40f76c in term 1. Re-sending same reply.
W20260812 08:02:59.817304 27067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49642 (request call id 2) took 61 ms (client timeout 81 ms). Trace:
I20260812 08:02:59.817603 27383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.817716 27383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:59.817360 27067 rpcz_store.cc:269] 0812 08:02:59.756186 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:59.756267 (+    81us) service_pool.cc:224] Handling call
0812 08:02:59.817290 (+ 61023us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260812 08:02:59.817799 27383 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:59.818096 27387 mvcc.cc:204] Tried to move back new op lower bound from 7317593210039336960 to 7317593209990176768. Current Snapshot: MvccSnapshot[applied={T|T < 7317593210039336960}]
I20260812 08:02:59.819415 27364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.819519 27364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 08:02:59.822530 27408 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:59.822609 27408 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:59.824097 27391 mvcc.cc:204] Tried to move back new op lower bound from 7317593210039336960 to 7317593209990176768. Current Snapshot: MvccSnapshot[applied={T|T < 7317593210039336960}]
I20260812 08:02:59.824146 27385 catalog_manager.cc:1383] Generated new cluster ID: 0a28dd6fae174da5846c058ea83a24b9
I20260812 08:02:59.824204 27385 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:59.824615 27414 mvcc.cc:204] Tried to move back new op lower bound from 7317593210039336960 to 7317593209990176768. Current Snapshot: MvccSnapshot[applied={T|T < 7317593210039336960}]
I20260812 08:02:59.825076 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9eb51c31866473582eec6326b40f76c. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.825203 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.825291 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.825690 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:59.833261 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9eb51c31866473582eec6326b40f76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:02:59.833388 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:59.841723 27385 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:59.842772 27385 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:59.873026 27385 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: Generated new TSK 0
I20260812 08:02:59.873680 27385 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:59.899977 23148 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0650HJ/build/debug/bin/kudu
/tmp/dist-test-task0650HJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.155.1:0
--local_ip_for_outbound_sockets=127.22.155.1
--webserver_interface=127.22.155.1
--webserver_port=0
--tserver_master_addrs=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--builtin_ntp_servers=127.22.155.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=25
--leader_failure_exp_backoff_max_delta_ms=1000
--heartbeat_interval_ms=25
--enable_txn_system_client_init=true with env {}
W20260812 08:03:00.053066 27424 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:00.053326 27424 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:00.053403 27424 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:00.058391 27424 flags.cc:432] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000
W20260812 08:03:00.058501 27424 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:00.058568 27424 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 08:03:00.058646 27424 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.155.1
I20260812 08:03:00.064106 27424 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.155.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_exp_backoff_max_delta_ms=1000
--raft_heartbeat_interval_ms=25
--fs_data_dirs=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.155.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.155.1
--webserver_port=0
--enable_txn_system_client_init=true
--heartbeat_interval_ms=25
--tserver_master_addrs=127.22.155.62:45729,127.22.155.61:42727,127.22.155.60:35659
--never_fsync=true
--heap_profile_path=/tmp/kudu.27424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.155.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:00.065639 27424 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:00.067188 27424 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:03:00.070782 27424 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 12
W20260812 08:03:00.075088 27439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:00.075127 27434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:00.075424 27435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:00.075601 27424 server_base.cc:1061] running on GCE node
I20260812 08:03:00.075973 27424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:00.076619 27424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:00.077881 27424 hybrid_clock.cc:648] HybridClock initialized: now 1786521780077839 us; error 57 us; skew 500 ppm
I20260812 08:03:00.080034 27424 webserver.cc:533] Webserver started at http://127.22.155.1:42657/ using document root <none> and password file <none>
I20260812 08:03:00.080790 27424 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:00.080883 27424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:00.081117 27424 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:00.082777 27424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/instance:
uuid: "e980b452b3e54a79a6c0ee01a630eec0"
format_stamp: "Formatted at 2026-08-12 08:03:00 on dist-test-slave-ffrd"
I20260812 08:03:00.083279 27424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance:
uuid: "e980b452b3e54a79a6c0ee01a630eec0"
format_stamp: "Formatted at 2026-08-12 08:03:00 on dist-test-slave-ffrd"
I20260812 08:03:00.086870 27424 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:03:00.089560 27450 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:00.090606 27424 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:03:00.090773 27424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data,/tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
uuid: "e980b452b3e54a79a6c0ee01a630eec0"
format_stamp: "Formatted at 2026-08-12 08:03:00 on dist-test-slave-ffrd"
I20260812 08:03:00.090883 27424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:00.105643 27424 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:00.106251 27424 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:00.106402 27424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:00.107973 27424 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:00.108026 27424 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:00.108067 27424 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:00.108134 27424 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:00.120103 27424 rpc_server.cc:307] RPC server started. Bound to: 127.22.155.1:37525
I20260812 08:03:00.120170 27655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.155.1:37525 every 8 connection(s)
I20260812 08:03:00.121285 27424 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/data/info.pb
I20260812 08:03:00.125587 27658 heartbeater.cc:344] Connected to a master server at 127.22.155.62:45729
I20260812 08:03:00.125802 27658 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:00.126319 27658 heartbeater.cc:507] Master 127.22.155.62:45729 requested a full tablet report, sending...
I20260812 08:03:00.127835 27047 ts_manager.cc:194] Registered new tserver with Master: e980b452b3e54a79a6c0ee01a630eec0 (127.22.155.1:37525)
I20260812 08:03:00.130273 27656 heartbeater.cc:344] Connected to a master server at 127.22.155.60:35659
I20260812 08:03:00.130374 27656 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:00.130637 27656 heartbeater.cc:507] Master 127.22.155.60:35659 requested a full tablet report, sending...
I20260812 08:03:00.130831 27659 heartbeater.cc:344] Connected to a master server at 127.22.155.61:42727
I20260812 08:03:00.130910 27659 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:00.131151 27659 heartbeater.cc:507] Master 127.22.155.61:42727 requested a full tablet report, sending...
I20260812 08:03:00.131249 23148 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0650HJ/build/debug/bin/kudu as pid 27424
I20260812 08:03:00.131358 23148 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0650HJ/test-tmp/client-negotiation-failover-itest.0.ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster.1786521736143602-23148-0/minicluster-data/ts-0/wal/instance
I20260812 08:03:00.131836 27271 ts_manager.cc:194] Registered new tserver with Master: e980b452b3e54a79a6c0ee01a630eec0 (127.22.155.1:37525)
I20260812 08:03:00.132230 27162 ts_manager.cc:194] Registered new tserver with Master: e980b452b3e54a79a6c0ee01a630eec0 (127.22.155.1:37525)
I20260812 08:03:00.133426 27271 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.155.1:42475
I20260812 08:03:00.155820 23148 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260812 08:03:00.160338 27656 heartbeater.cc:499] Master 127.22.155.60:35659 was elected leader, sending a full tablet report...
W20260812 08:03:00.187949 27310 generic_service.cc:139] {username='slave'} at 127.0.0.1:39652 forcing change of non-runtime-safe flag rpc_negotiation_inject_delay_ms
I20260812 08:03:00.188107 27310 generic_service.cc:175] {username='slave'} at 127.0.0.1:39652 changed flags via RPC: rpc_negotiation_inject_delay_ms from '0' to '6000'
W20260812 08:03:00.189616 27328 negotiation.cc:265] Injecting 6000ms delay in negotiation
W20260812 08:03:00.705289 27671 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:03:00.189115 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:35659 (local address 127.0.0.1:39654)
0812 08:03:00.189239 (+   124us) negotiation.cc:107] Waiting for socket to connect
0812 08:03:00.189256 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 08:03:00.189332 (+    76us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:03:00.705181 (+515849us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.60:35659: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 08:03:00.705832 27665 master_proxy_rpc.cc:203] Re-attempting ListTables request to leader Master (127.22.155.60:35659)
W20260812 08:03:00.706960 27690 negotiation.cc:265] Injecting 6000ms delay in negotiation
I20260812 08:03:00.784168 27379 catalog_manager.cc:1295] Loaded cluster ID: 0a28dd6fae174da5846c058ea83a24b9
I20260812 08:03:00.784255 27379 catalog_manager.cc:1588] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: loading cluster ID for follower catalog manager: success
I20260812 08:03:00.786042 27379 catalog_manager.cc:1610] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: acquiring CA information for follower catalog manager: success
I20260812 08:03:00.787119 27379 catalog_manager.cc:1638] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:03:00.823729 27408 catalog_manager.cc:1295] Loaded cluster ID: 0a28dd6fae174da5846c058ea83a24b9
I20260812 08:03:00.823829 27408 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: loading cluster ID for follower catalog manager: success
I20260812 08:03:00.825547 27408 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: acquiring CA information for follower catalog manager: success
I20260812 08:03:00.826658 27408 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 08:03:01.217396 27689 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:03:00.706438 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:35659 (local address 127.0.0.1:39662)
0812 08:03:00.706891 (+   453us) negotiation.cc:107] Waiting for socket to connect
0812 08:03:00.706905 (+    14us) client_negotiation.cc:175] Beginning negotiation
0812 08:03:00.706996 (+    91us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:03:01.217179 (+510183us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.60:35659: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":53,"threads_started":1}
W20260812 08:03:01.219851 27697 negotiation.cc:265] Injecting 6000ms delay in negotiation
W20260812 08:03:01.729288 27695 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:03:01.219457 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:35659 (local address 127.0.0.1:39668)
0812 08:03:01.221468 (+  2011us) negotiation.cc:107] Waiting for socket to connect
0812 08:03:01.221483 (+    15us) client_negotiation.cc:175] Beginning negotiation
0812 08:03:01.221565 (+    82us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:03:01.729170 (+507605us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.60:35659: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":1929,"thread_start_us":60,"threads_started":1}
I20260812 08:03:01.748237 27712 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9eb51c31866473582eec6326b40f76c)
I20260812 08:03:01.748407 27712 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:03:01.749382 27712 leader_election.cc:290] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbe07b2b9c554d7ebfb03d50c76d3c2a (127.22.155.61:42727), a9eb51c31866473582eec6326b40f76c (127.22.155.60:35659)
I20260812 08:03:01.749795 27176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" is_pre_election: true
I20260812 08:03:01.749928 27176 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fcf06aba4f947109641fc459b340016 in term 1.
I20260812 08:03:01.750258 27028 leader_election.cc:304] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fcf06aba4f947109641fc459b340016, bbe07b2b9c554d7ebfb03d50c76d3c2a; no voters: 
I20260812 08:03:01.750511 27712 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:03:01.750582 27712 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a9eb51c31866473582eec6326b40f76c)
I20260812 08:03:01.750613 27712 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:03:01.751484 27712 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:03:01.751858 27712 leader_election.cc:290] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 election: Requested vote from peers bbe07b2b9c554d7ebfb03d50c76d3c2a (127.22.155.61:42727), a9eb51c31866473582eec6326b40f76c (127.22.155.60:35659)
I20260812 08:03:01.752131 27176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a"
I20260812 08:03:01.752267 27176 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:03:01.753129 27176 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fcf06aba4f947109641fc459b340016 in term 2.
I20260812 08:03:01.753418 27028 leader_election.cc:304] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fcf06aba4f947109641fc459b340016, bbe07b2b9c554d7ebfb03d50c76d3c2a; no voters: 
I20260812 08:03:01.753594 27712 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 08:03:01.753747 27712 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 2 LEADER]: Becoming Leader. State: Replica: 6fcf06aba4f947109641fc459b340016, State: Running, Role: LEADER
I20260812 08:03:01.754055 27712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:03:01.755215 27716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fcf06aba4f947109641fc459b340016. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:01.755326 27716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:01.755682 27718 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:01.756536 27718 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:01.756918 27718 catalog_manager.cc:1295] Loaded cluster ID: 0a28dd6fae174da5846c058ea83a24b9
I20260812 08:03:01.757020 27718 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:01.757565 27718 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:01.757968 27718 catalog_manager.cc:6103] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016: Loaded TSK: 0
I20260812 08:03:01.758560 27718 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:03:01.774771 27176 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 2 FOLLOWER]: Refusing update from remote peer 6fcf06aba4f947109641fc459b340016: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 08:03:01.775252 27716 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 08:03:01.776376 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fcf06aba4f947109641fc459b340016. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:01.776513 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:03:01.777655 27716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:01.777762 27716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:01.777941 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:01.778049 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:03:01.795025 27658 heartbeater.cc:499] Master 127.22.155.62:45729 was elected leader, sending a full tablet report...
W20260812 08:03:01.911010 27023 leader_election.cc:336] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9eb51c31866473582eec6326b40f76c (127.22.155.60:35659): Timed out: RequestConsensusVote RPC to 127.22.155.60:35659 timed out after 0.080s (SENT)
W20260812 08:03:01.911336 27023 leader_election.cc:336] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a9eb51c31866473582eec6326b40f76c (127.22.155.60:35659): Timed out: RequestConsensusVote RPC to 127.22.155.60:35659 timed out after 0.079s (SENT)
W20260812 08:03:02.241308 27706 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:03:01.733767 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:35659 (local address 127.0.0.1:39680)
0812 08:03:01.734015 (+   248us) negotiation.cc:107] Waiting for socket to connect
0812 08:03:01.734028 (+    13us) client_negotiation.cc:175] Beginning negotiation
0812 08:03:01.734105 (+    77us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:03:02.241182 (+507077us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.155.60:35659: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":58,"threads_started":1}
I20260812 08:03:02.256613 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 27424
I20260812 08:03:02.257782 27280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a9eb51c31866473582eec6326b40f76c" is_pre_election: true
I20260812 08:03:02.258339 27281 raft_consensus.cc:3055] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 LEADER]: Stepping down as leader of term 1
I20260812 08:03:02.258423 27281 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9eb51c31866473582eec6326b40f76c, State: Running, Role: LEADER
I20260812 08:03:02.257786 27284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a9eb51c31866473582eec6326b40f76c" is_pre_election: true
I20260812 08:03:02.258625 27281 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } }
I20260812 08:03:02.258766 27281 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:03:02.259106 27067 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [term 2 LEADER]: Rejecting Update request from peer a9eb51c31866473582eec6326b40f76c for earlier term 1. Current term is 2. Ops: []
I20260812 08:03:02.259981 27281 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 2 FOLLOWER]: Refusing update from remote peer 6fcf06aba4f947109641fc459b340016: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260812 08:03:02.262446 27357 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:03:02.263010 27279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a9eb51c31866473582eec6326b40f76c"
W20260812 08:03:02.263521 27735 negotiation.cc:265] Injecting 6000ms delay in negotiation
I20260812 08:03:02.263965 27716 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6fcf06aba4f947109641fc459b340016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260812 08:03:02.264394 27279 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fcf06aba4f947109641fc459b340016 in term 2.
I20260812 08:03:02.265020 27283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fcf06aba4f947109641fc459b340016" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a9eb51c31866473582eec6326b40f76c"
I20260812 08:03:02.265494 27691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fcf06aba4f947109641fc459b340016. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:02.265609 27691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:03:02.265707 27691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6fcf06aba4f947109641fc459b340016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fcf06aba4f947109641fc459b340016" member_type: VOTER last_known_addr { host: "127.22.155.62" port: 45729 } } peers { permanent_uuid: "bbe07b2b9c554d7ebfb03d50c76d3c2a" member_type: VOTER last_known_addr { host: "127.22.155.61" port: 42727 } } peers { permanent_uuid: "a9eb51c31866473582eec6326b40f76c" member_type: VOTER last_known_addr { host: "127.22.155.60" port: 35659 } } }
I20260812 08:03:02.265805 27691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eb51c31866473582eec6326b40f76c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:03:02.266278 27176 raft_consensus.cc:1240] T 00000000000000000000000000000000 P bbe07b2b9c554d7ebfb03d50c76d3c2a [term 2 FOLLOWER]: Rejecting Update request from peer a9eb51c31866473582eec6326b40f76c for earlier term 1. Current term is 2. Ops: []
W20260812 08:03:02.269071 27738 negotiation.cc:265] Injecting 6000ms delay in negotiation
W20260812 08:03:02.270799 27250 connection.cc:573] server connection from 127.22.155.1:42475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:03:02.271490 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 26994
W20260812 08:03:02.290629 27140 connection.cc:573] server connection from 127.0.0.1:52744 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:03:02.291312 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 27115
I20260812 08:03:02.304904 23148 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0650HJ/build/debug/bin/kudu with pid 27220
W20260812 08:03:02.318271 27733 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:03:02.248498 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.22.155.60:35659 (local address 127.0.0.1:39688)
0812 08:03:02.248752 (+   254us) negotiation.cc:107] Waiting for socket to connect
0812 08:03:02.248768 (+    16us) client_negotiation.cc:175] Beginning negotiation
0812 08:03:02.248878 (+   110us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 08:03:02.318082 (+ 69204us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.155.60:35659: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":164,"spinlock_wait_cycles":13696,"thread_start_us":78,"threads_started":1}
2026-08-12T08:03:02Z chronyd exiting
[       OK ] ClientFailoverOnNegotiationTimeoutITest.Kudu2021NegotiateWithMaster (3174 ms)
[----------] 4 tests from ClientFailoverOnNegotiationTimeoutITest (46179 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (46179 ms total)
[  PASSED  ] 4 tests.
