Note: This is test shard 6 of 6.
[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from RaftConsensusITest
[ RUN      ] RaftConsensusITest.InsertUniqueKeysWithCrashyNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 06:36:51.960366  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:36:51.960546  8019 ts_itest-base.cc:116] --------------
I20260812 06:36:51.960559  8019 ts_itest-base.cc:117] 7 tablet servers
I20260812 06:36:51.960572  8019 ts_itest-base.cc:118] 7 replicas per TS
I20260812 06:36:51.960583  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:36:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:51Z Disabled control of system clock
I20260812 06:36:51.973654  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:33393
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:33393 with env {}
W20260812 06:36:52.089355  8037 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:52.089562  8037 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:52.089600  8037 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:52.092849  8037 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:52.092914  8037 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:52.092934  8037 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:52.092952  8037 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:52.096738  8037 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:33393
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:52.097857  8037 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:52.099179  8037 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:52.105820  8051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.106496  8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.106540  8037 server_base.cc:1061] running on GCE node
W20260812 06:36:52.106475  8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.107246  8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:52.108199  8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:52.109357  8037 hybrid_clock.cc:648] HybridClock initialized: now 1786516612109248 us; error 158 us; skew 500 ppm
I20260812 06:36:52.111425  8037 webserver.cc:533] Webserver started at http://127.7.212.254:39245/ using document root <none> and password file <none>
I20260812 06:36:52.112043  8037 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:52.112131  8037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:52.112413  8037 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:52.114051  8037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/instance:
uuid: "8b8d1844065f4ff88f8e66e499a4d51b"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.114573  8037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal/instance:
uuid: "8b8d1844065f4ff88f8e66e499a4d51b"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.118013  8037 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:52.120412  8060 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.121573  8037 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:52.121754  8037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
uuid: "8b8d1844065f4ff88f8e66e499a4d51b"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.121862  8037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:52.136466  8037 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:52.137202  8037 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:52.137354  8037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:52.146173  8037 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:33393
I20260812 06:36:52.146171  8147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:33393 every 8 connection(s)
I20260812 06:36:52.147372  8037 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
I20260812 06:36:52.150668  8148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:52.150871  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 8037
I20260812 06:36:52.151019  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal/instance
I20260812 06:36:52.159072  8148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b: Bootstrap starting.
I20260812 06:36:52.161942  8148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:52.162889  8148 log.cc:826] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:52.165153  8148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b: No bootstrap required, opened a new log
I20260812 06:36:52.168668  8148 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } }
I20260812 06:36:52.169028  8148 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:52.169143  8148 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8d1844065f4ff88f8e66e499a4d51b, State: Initialized, Role: FOLLOWER
I20260812 06:36:52.169628  8148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } }
I20260812 06:36:52.169759  8148 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:36:52.169854  8148 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:36:52.170022  8148 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:52.171183  8148 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } }
I20260812 06:36:52.171608  8148 leader_election.cc:304] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8d1844065f4ff88f8e66e499a4d51b; no voters: 
I20260812 06:36:52.171959  8148 leader_election.cc:290] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:36:52.172138  8159 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:52.172386  8159 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [term 1 LEADER]: Becoming Leader. State: Replica: 8b8d1844065f4ff88f8e66e499a4d51b, State: Running, Role: LEADER
I20260812 06:36:52.172748  8159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } }
I20260812 06:36:52.173288  8148 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:52.175205  8164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8d1844065f4ff88f8e66e499a4d51b. Latest consensus state: current_term: 1 leader_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } } }
I20260812 06:36:52.175920  8164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:52.176153  8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8d1844065f4ff88f8e66e499a4d51b" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 33393 } } }
I20260812 06:36:52.176255  8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:52.176360  8183 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:52.179312  8183 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:52.185127  8183 catalog_manager.cc:1383] Generated new cluster ID: 68292e0dcc0a45b7a049e77844b713d9
I20260812 06:36:52.185245  8183 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:52.197638  8183 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:52.198560  8183 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:52.206324  8183 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 8b8d1844065f4ff88f8e66e499a4d51b: Generated new TSK 0
I20260812 06:36:52.207021  8183 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:36:52.216144  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:52.328462  8188 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:52.328704  8188 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:52.328738  8188 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:52.328786  8188 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:52.328855  8188 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:52.328917  8188 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:52.328984  8188 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:52.332263  8188 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:52.332423  8188 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:36:52.336244  8188 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.8188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:52.337426  8188 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:52.338554  8188 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:52.341308  8188 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:52.345291  8200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.345306  8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.345712  8198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.346133  8188 server_base.cc:1061] running on GCE node
I20260812 06:36:52.346596  8188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:52.347137  8188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:52.348328  8188 hybrid_clock.cc:648] HybridClock initialized: now 1786516612348310 us; error 37 us; skew 500 ppm
I20260812 06:36:52.350193  8188 webserver.cc:533] Webserver started at http://127.7.212.193:33451/ using document root <none> and password file <none>
I20260812 06:36:52.350756  8188 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:52.350838  8188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:52.351058  8188 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:52.352612  8188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/instance:
uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.353179  8188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal/instance:
uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.356472  8188 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:36:52.358944  8213 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.360167  8188 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:52.360324  8188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.360438  8188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:52.374302  8188 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:52.375037  8188 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:52.375258  8188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:52.375845  8188 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:52.377034  8188 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:52.377116  8188 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.377190  8188 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:52.377243  8188 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.390372  8188 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:37797
I20260812 06:36:52.390452  8418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:37797 every 8 connection(s)
I20260812 06:36:52.391446  8188 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
I20260812 06:36:52.393746  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 8188
I20260812 06:36:52.393862  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal/instance
I20260812 06:36:52.396391  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:36:52.401592  8420 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:52.401880  8420 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:52.402436  8420 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:52.404197  8085 ts_manager.cc:194] Registered new tserver with Master: 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:52.405447  8085 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:48329
W20260812 06:36:52.517683  8428 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:52.518070  8428 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:52.518150  8428 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:52.518218  8428 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:52.518318  8428 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:52.518405  8428 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:52.518517  8428 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:52.524242  8428 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:52.524513  8428 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:36:52.532719  8428 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.8428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:52.534394  8428 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:52.536226  8428 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:52.540053  8428 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:52.543944  8436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.543998  8433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.544030  8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.544344  8428 server_base.cc:1061] running on GCE node
I20260812 06:36:52.544881  8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:52.545414  8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:52.546588  8428 hybrid_clock.cc:648] HybridClock initialized: now 1786516612546577 us; error 34 us; skew 500 ppm
I20260812 06:36:52.548246  8428 webserver.cc:533] Webserver started at http://127.7.212.194:41195/ using document root <none> and password file <none>
I20260812 06:36:52.548868  8428 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:52.548954  8428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:52.549151  8428 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:52.550719  8428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/instance:
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.551239  8428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal/instance:
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.554641  8428 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:52.556944  8451 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.557914  8428 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:52.558029  8428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.558121  8428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:52.576058  8428 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:52.577172  8428 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:52.577397  8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:52.577979  8428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:52.579061  8428 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:52.579130  8428 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.579196  8428 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:52.579247  8428 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.591271  8661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:34921 every 8 connection(s)
I20260812 06:36:52.591251  8428 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:34921
I20260812 06:36:52.592255  8428 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
I20260812 06:36:52.596418  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 8428
I20260812 06:36:52.596537  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal/instance
I20260812 06:36:52.599464  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:36:52.602547  8663 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:52.602802  8663 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:52.603335  8663 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:52.604532  8085 ts_manager.cc:194] Registered new tserver with Master: 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921)
I20260812 06:36:52.605399  8085 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:39637
W20260812 06:36:52.712198  8669 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:52.712390  8669 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:52.712409  8669 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:52.712423  8669 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:52.712458  8669 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:52.712477  8669 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:52.712527  8669 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:52.715775  8669 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:52.715883  8669 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:36:52.720202  8669 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.8669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:52.721324  8669 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:52.722458  8669 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:52.725303  8669 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:52.729161  8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.729216  8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.729192  8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.729441  8669 server_base.cc:1061] running on GCE node
I20260812 06:36:52.729832  8669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:52.730404  8669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:52.731564  8669 hybrid_clock.cc:648] HybridClock initialized: now 1786516612731523 us; error 54 us; skew 500 ppm
I20260812 06:36:52.733444  8669 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:52.733999  8669 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:52.734064  8669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:52.734263  8669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:52.735782  8669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/instance:
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.736356  8669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal/instance:
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.739953  8669 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:52.742228  8695 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.743417  8669 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:52.743513  8669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.743642  8669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:52.762456  8669 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:52.763180  8669 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:52.763389  8669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:52.764010  8669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:52.765079  8669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:52.765156  8669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.765235  8669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:52.765280  8669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.777316  8669 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:36:52.777345  8901 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:52.778367  8669 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:52.787488  8902 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:52.787729  8902 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:52.787847  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 8669
I20260812 06:36:52.787964  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal/instance
I20260812 06:36:52.788304  8902 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:52.789388  8083 ts_manager.cc:194] Registered new tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:52.789985  8083 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:58161
I20260812 06:36:52.790266  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.196:0
--local_ip_for_outbound_sockets=127.7.212.196
--webserver_interface=127.7.212.196
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:52.899411  8905 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:52.899605  8905 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:52.899631  8905 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:52.899646  8905 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:52.899682  8905 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:52.899711  8905 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:52.899756  8905 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:52.902976  8905 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:52.903074  8905 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.196
I20260812 06:36:52.906809  8905 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.7.212.196
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.8905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.196

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:52.907737  8905 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:52.908809  8905 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:52.911413  8905 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:52.914903  8922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.914952  8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:52.914892  8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:52.915426  8905 server_base.cc:1061] running on GCE node
I20260812 06:36:52.915797  8905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:52.916373  8905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:52.917555  8905 hybrid_clock.cc:648] HybridClock initialized: now 1786516612917538 us; error 34 us; skew 500 ppm
I20260812 06:36:52.919154  8905 webserver.cc:533] Webserver started at http://127.7.212.196:40525/ using document root <none> and password file <none>
I20260812 06:36:52.919692  8905 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:52.919747  8905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:52.919952  8905 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:52.921557  8905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/instance:
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.922057  8905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal/instance:
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.925333  8905 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:36:52.927485  8931 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.928563  8905 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:52.928695  8905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:52.928817  8905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:52.951607  8905 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:52.952287  8905 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:52.952478  8905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:52.953225  8905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:52.954332  8905 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:52.954412  8905 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.954484  8905 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:52.954527  8905 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:52.969599  8905 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.196:38991
I20260812 06:36:52.969619  9127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.196:38991 every 8 connection(s)
I20260812 06:36:52.970744  8905 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 06:36:52.978693  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 8905
I20260812 06:36:52.978829  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal/instance
I20260812 06:36:52.979627  9129 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:52.979882  9129 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:52.980422  9129 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:52.981469  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.197:0
--local_ip_for_outbound_sockets=127.7.212.197
--webserver_interface=127.7.212.197
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:36:52.981544  8083 ts_manager.cc:194] Registered new tserver with Master: 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991)
I20260812 06:36:52.982118  8083 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.196:48777
W20260812 06:36:53.090350  9134 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:53.090571  9134 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:53.090603  9134 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:53.090641  9134 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:53.090711  9134 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:53.090759  9134 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:53.090831  9134 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:53.094390  9134 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:53.094565  9134 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.197
I20260812 06:36:53.098989  9134 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.7.212.197
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.197

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:53.100159  9134 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:53.101526  9134 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:53.104362  9134 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:53.108143  9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.108202  9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.108143  9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:53.108492  9134 server_base.cc:1061] running on GCE node
I20260812 06:36:53.109166  9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:53.109797  9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:53.111018  9134 hybrid_clock.cc:648] HybridClock initialized: now 1786516613110995 us; error 44 us; skew 500 ppm
I20260812 06:36:53.113062  9134 webserver.cc:533] Webserver started at http://127.7.212.197:33771/ using document root <none> and password file <none>
I20260812 06:36:53.113646  9134 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:53.113731  9134 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:53.113951  9134 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:53.115535  9134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/instance:
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.116015  9134 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal/instance:
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.119372  9134 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 06:36:53.121589  9161 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.122768  9134 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:53.122910  9134 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.123019  9134 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:53.150941  9134 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:53.151634  9134 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:53.151844  9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:53.152508  9134 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:53.153687  9134 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:53.153769  9134 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.153839  9134 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:53.153888  9134 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.168803  9134 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.197:41571
I20260812 06:36:53.168825  9359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.197:41571 every 8 connection(s)
I20260812 06:36:53.169867  9134 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 06:36:53.179507  9360 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:53.179714  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 9134
I20260812 06:36:53.179776  9360 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:53.179836  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal/instance
I20260812 06:36:53.180301  9360 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:53.181497  8083 ts_manager.cc:194] Registered new tserver with Master: a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571)
I20260812 06:36:53.182071  8083 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.197:34671
I20260812 06:36:53.182199  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.198:0
--local_ip_for_outbound_sockets=127.7.212.198
--webserver_interface=127.7.212.198
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:53.299790  9365 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:53.300004  9365 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:53.300025  9365 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:53.300042  9365 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:53.300082  9365 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:53.300101  9365 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:53.300146  9365 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:53.303911  9365 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:53.304023  9365 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.198
I20260812 06:36:53.308960  9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.7.212.198
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.198

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:53.310065  9365 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:53.311190  9365 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:53.314219  9365 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:53.318543  9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.318570  9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.318568  9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:53.319036  9365 server_base.cc:1061] running on GCE node
I20260812 06:36:53.319427  9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:53.320123  9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:53.321310  9365 hybrid_clock.cc:648] HybridClock initialized: now 1786516613321284 us; error 39 us; skew 500 ppm
I20260812 06:36:53.323093  9365 webserver.cc:533] Webserver started at http://127.7.212.198:35371/ using document root <none> and password file <none>
I20260812 06:36:53.323686  9365 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:53.323745  9365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:53.323943  9365 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:53.325937  9365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/instance:
uuid: "d12989a6d0f445359c9b09b7ce6cb578"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.326514  9365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal/instance:
uuid: "d12989a6d0f445359c9b09b7ce6cb578"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.330864  9365 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:36:53.333246  9385 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.334301  9365 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:53.334422  9365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "d12989a6d0f445359c9b09b7ce6cb578"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.334528  9365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:53.351610  9365 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:53.352470  9365 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:53.352742  9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:53.353503  9365 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:53.354825  9365 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:53.354882  9365 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.354954  9365 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:53.355001  9365 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.370378  9578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.198:35139 every 8 connection(s)
I20260812 06:36:53.370376  9365 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.198:35139
I20260812 06:36:53.371459  9365 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 06:36:53.380314  9579 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:53.380537  9579 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:53.380815  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 9365
I20260812 06:36:53.380929  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal/instance
I20260812 06:36:53.381181  9579 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:53.382477  8085 ts_manager.cc:194] Registered new tserver with Master: d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139)
I20260812 06:36:53.383142  8085 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.198:48021
I20260812 06:36:53.383498  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.199:0
--local_ip_for_outbound_sockets=127.7.212.199
--webserver_interface=127.7.212.199
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:36:53.407526  8420 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
W20260812 06:36:53.492147  9583 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:53.492394  9583 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:53.492426  9583 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:53.492494  9583 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:53.492573  9583 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:53.492622  9583 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:53.492702  9583 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:53.495972  9583 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:53.496137  9583 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.199
I20260812 06:36:53.499926  9583 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.7.212.199
--webserver_port=0
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.9583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.199

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:53.500986  9583 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:53.502105  9583 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:53.504879  9583 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:53.508836  9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.508859  9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:53.508826  9592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:53.509076  9583 server_base.cc:1061] running on GCE node
I20260812 06:36:53.509603  9583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:53.510156  9583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:53.511355  9583 hybrid_clock.cc:648] HybridClock initialized: now 1786516613511337 us; error 31 us; skew 500 ppm
I20260812 06:36:53.513424  9583 webserver.cc:533] Webserver started at http://127.7.212.199:34823/ using document root <none> and password file <none>
I20260812 06:36:53.514041  9583 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:53.514106  9583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:53.514379  9583 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:53.515946  9583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/instance:
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.516461  9583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal/instance:
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.519979  9583 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 06:36:53.522317  9609 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.523268  9583 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:53.523365  9583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:53.523432  9583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:53.546046  9583 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:53.546736  9583 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:53.546928  9583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:53.547506  9583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:53.548564  9583 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:53.548614  9583 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.548651  9583 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:53.548671  9583 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:53.559458  9583 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.199:34285
I20260812 06:36:53.559486  9813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.199:34285 every 8 connection(s)
I20260812 06:36:53.560442  9583 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 06:36:53.561112  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 9583
I20260812 06:36:53.561228  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal/instance
I20260812 06:36:53.569216  9815 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:53.569486  9815 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:53.570067  9815 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:53.571215  8085 ts_manager.cc:194] Registered new tserver with Master: c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:53.571872  8085 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.199:44791
I20260812 06:36:53.576869  8019 external_mini_cluster.cc:955] 7 TS(s) registered with all masters
I20260812 06:36:53.593554  8085 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33498:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260812 06:36:53.594913  8085 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20260812 06:36:53.608248  8663 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.614229  8541 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.614262  9240 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.615270  9240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.615270  8541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.619122  8289 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.619130  9476 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.620111  9476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.620138  8289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.624294  9695 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.625250  9695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.628001  9013 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.629225  9013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.633741  9857 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap starting.
I20260812 06:36:53.635011  9859 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap starting.
I20260812 06:36:53.634924  8778 tablet_service.cc:1511] Processing CreateTablet for tablet 2cd3bf4d446147de95e618239dd3074c (DEFAULT_TABLE table=test-workload [id=383db6ea515e4292abfa510513502daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:36:53.635840  9865 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap starting.
I20260812 06:36:53.636065  8778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd3bf4d446147de95e618239dd3074c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:53.636370  9857 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.637198  9859 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.637429  9857 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.638008  9859 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.638042  9865 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.638402  9864 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap starting.
I20260812 06:36:53.638813  9865 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.639796  9856 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Bootstrap starting.
I20260812 06:36:53.640705  9859 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: No bootstrap required, opened a new log
I20260812 06:36:53.638926  9860 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap starting.
W20260812 06:36:53.640967  9859 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 06:36:53.641928  9864 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.642577  9865 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: No bootstrap required, opened a new log
I20260812 06:36:53.642807  9865 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20260812 06:36:53.642786  9864 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.642874  9860 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.643671  9860 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.644632  9857 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: No bootstrap required, opened a new log
W20260812 06:36:53.644835  9857 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 129ms of latency in SegmentAllocator::Sync()
I20260812 06:36:53.645757  9856 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.646476  9860 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: No bootstrap required, opened a new log
I20260812 06:36:53.646608  9864 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: No bootstrap required, opened a new log
W20260812 06:36:53.646724  9860 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 91ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.646792  9864 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 48ms of latency in SegmentAllocator::Sync()
I20260812 06:36:53.646728  9856 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.646711  9865 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.646979  9865 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.647046  9865 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.647194  9873 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
I20260812 06:36:53.647554  9865 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.648641  9865 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260812 06:36:53.649075  9129 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.649379  9856 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: No bootstrap required, opened a new log
W20260812 06:36:53.649528  9856 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 19ms of latency in SegmentAllocator::Sync()
I20260812 06:36:53.649555  9873 tablet_bootstrap.cc:654] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:53.650223  9873 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:53.651584  9873 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: No bootstrap required, opened a new log
I20260812 06:36:53.651810  9873 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20260812 06:36:53.654386  9873 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.654569  9873 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.654644  9873 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.655055  9873 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.655718  8902 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.655921  9873 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.002s
I20260812 06:36:53.668888  9856 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.001s
I20260812 06:36:53.671926  9856 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.672243  9856 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.672333  9856 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8291da04bf4d8dafd875d0dfe291f0, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.672891  9856 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.673842  9856 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
I20260812 06:36:53.677512  9859 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent bootstrapping tablet: real 0.043s	user 0.005s	sys 0.000s
I20260812 06:36:53.680174  9859 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.680419  9859 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.680503  9859 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.680956  9859 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.681738  9815 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.681905  9859 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260812 06:36:53.695067  9864 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent bootstrapping tablet: real 0.057s	user 0.005s	sys 0.000s
I20260812 06:36:53.697717  9864 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.697916  9864 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.697978  9864 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.698458  9864 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.699236  9360 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.699338  9864 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260812 06:36:53.705014  9881 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:53.705281  9881 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.707026  9881 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:53.710564  9885 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:53.710810  9885 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.714206  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:53.714496  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 0.
I20260812 06:36:53.713845  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:36:53.716142  8697 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:53.716725  9885 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:53.717705  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:53.717900  9734 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 0.
I20260812 06:36:53.719461  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:53.719724  9044 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 0.
I20260812 06:36:53.720158  8697 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:53.720467  9881 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:53.720551  9881 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:53.720611  9881 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.721544  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:53.721846  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 0.
I20260812 06:36:53.722065  9881 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:53.722275  9132 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:36:53.722723  9881 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:53.722252  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:53.723423  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:53.723553  9734 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.724578  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:53.724781  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:53.724809  9734 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 1.
I20260812 06:36:53.724819  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:53.724905  9044 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.724961  9272 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 0 FOLLOWER]: Advancing to term 1
W20260812 06:36:53.725342  8701 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:53.725721  8701 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:53.725961  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:53.725960  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:53.726068  9044 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 1.
I20260812 06:36:53.726087  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.727380  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 1.
W20260812 06:36:53.727773  8697 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:53.727931  8697 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:53.728305  9881 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:53.728672  9881 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:36:53.728703  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 1.
I20260812 06:36:53.728791  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:53.728972  9044 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 1: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:53.729130  8810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:53.729110  9881 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.732443  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
W20260812 06:36:53.733036  9617 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:53.733137  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:53.733345  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 1: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:53.734054  9611 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10
W20260812 06:36:53.734309  9924 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 46ms of latency in SegmentAllocator::Sync()
I20260812 06:36:53.734393  9885 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 06:36:53.734285  8083 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 1 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:53.734746  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:53.734961  9272 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 1: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:53.736176  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:36:53.736624  9611 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:53.737998  9860 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent bootstrapping tablet: real 0.099s	user 0.005s	sys 0.000s
I20260812 06:36:53.740336  9860 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.740499  9860 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.740564  9860 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12989a6d0f445359c9b09b7ce6cb578, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.741093  9860 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.741813  9579 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:53.741962  9860 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20260812 06:36:53.774116  9857 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent bootstrapping tablet: real 0.141s	user 0.006s	sys 0.000s
I20260812 06:36:53.777261  9857 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.777486  9857 raft_consensus.cc:385] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:53.777536  9857 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Initialized, Role: FOLLOWER
I20260812 06:36:53.778055  9857 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:53.778982  9857 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
W20260812 06:36:53.782305  8903 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:53.782932  9924 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.783200  9881 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:53.783241  9881 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:53.784274  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.784677  9734 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.785068  9923 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:53.785712  9044 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.786445  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.786578  8577 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:53.787719  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.787727  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.788269  9881 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:53.788677  9881 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:53.789072  9881 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:53.789304  9272 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:53.792861  9881 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:53.793114  9923 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 06:36:53.800473  9964 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 17ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.809062  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 48ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.809521  9966 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 19ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.811362  9816 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:53.812346  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 44ms of latency in SegmentAllocator::Sync()
W20260812 06:36:53.813756  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 121ms of latency in SegmentAllocator::Sync()
E20260812 06:36:53.829849  9953 fault_injection.cc:59] injecting fault for kudu (pid 8669): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:53.844697  8667 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:53.859763  9397 connection.cc:573] server connection from 127.7.212.195:52583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:53.859815  9818 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613) has failed: Network error: recv got EOF from 127.7.212.195:40613 (error 108)
W20260812 06:36:53.860270  8460 connection.cc:573] server connection from 127.7.212.195:38703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:53.865044  8019 ts_itest-base.cc:575] TS 2 appears to have crashed. Restarting.
I20260812 06:36:53.865587  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:53.869586  8477 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:53.869586  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:53.872581  9581 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:53.893548  8421 tablet.cc:2406] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:53.925848  8973 tablet_service.cc:3018] Rejecting scan request for tablet 2cd3bf4d446147de95e618239dd3074c: Uninitialized: clean time has not yet been initialized
W20260812 06:36:53.927412  8971 tablet_service.cc:3018] Rejecting scan request for tablet 2cd3bf4d446147de95e618239dd3074c: Uninitialized: clean time has not yet been initialized
W20260812 06:36:53.932875  8255 tablet_service.cc:3018] Rejecting scan request for tablet 2cd3bf4d446147de95e618239dd3074c: Uninitialized: clean time has not yet been initialized
W20260812 06:36:53.932909  8256 tablet_service.cc:3018] Rejecting scan request for tablet 2cd3bf4d446147de95e618239dd3074c: Uninitialized: clean time has not yet been initialized
W20260812 06:36:53.935422  8932 connection.cc:466] server connection from 127.7.212.195:59333 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:59333 (request call id 3) could send its response
W20260812 06:36:53.996129  9968 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:53.996416  9968 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:53.996455  9968 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:53.996496  9968 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:53.996578  9968 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:53.996654  9968 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:53.996752  9968 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:54.000597  9968 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:54.000816  9968 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:36:54.005165  9968 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.9968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:54.006377  9968 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:54.007635  9968 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:54.010759  9968 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:54.014709  9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:54.014726  9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:54.014827  9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:54.015275  9968 server_base.cc:1061] running on GCE node
I20260812 06:36:54.015733  9968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:54.016546  9968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:54.017753  9968 hybrid_clock.cc:648] HybridClock initialized: now 1786516614017732 us; error 41 us; skew 500 ppm
I20260812 06:36:54.019629  9968 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:54.020316  9968 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:54.020424  9968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:54.024171  9968 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:36:54.026340  9994 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:54.027400  9968 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:54.027494  9968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:54.027995  9968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:54.089641  9968 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:54.090349  9968 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:54.090507  9968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:54.091087  9968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:54.091321  9884 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.091488  9884 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.092499 10011 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:54.093253  9884 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:54.095309  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.095474  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:54.095935  8214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:36:54.097189  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.098006  9968 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:54.098119  9968 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20260812 06:36:54.098245  9968 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:54.100103  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.100310  8577 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 for term 2 because replica has last-logged OpId of term: 1 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260812 06:36:54.100428  9968 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:54.100521  9968 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:54.101308 10011 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
I20260812 06:36:54.103255  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.104087  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:54.105237  8453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:36:54.106920  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.107467  8217 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0; no voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:54.107776  9884 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 06:36:54.108248  8214 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:36:54.108871  9925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.109030  9925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.109757 10011 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:54.110474  9925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.113729  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.114761  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.117367  9968 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:36:54.118765  9968 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:54.119136  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.119292  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 1.
I20260812 06:36:54.119575  9885 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.119684  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.119710  9885 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:54.120141  8453 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:36:54.120434  9885 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.120644  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.120864  9272 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 for term 2 because replica has last-logged OpId of term: 1 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 9.
I20260812 06:36:54.121304  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.121569  9949 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 06:36:54.121538  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.121680  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 1.
I20260812 06:36:54.121996  9892 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.122102  9892 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.122848  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.122987  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 1.
I20260812 06:36:54.123242  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.123426  9510 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 for term 2 because replica has last-logged OpId of term: 1 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 9.
I20260812 06:36:54.123708  9892 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.123816  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.126048  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 9968
I20260812 06:36:54.126062  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.126200  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 1.
I20260812 06:36:54.127249  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.127382  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 1.
I20260812 06:36:54.128017  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.128130  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 1.
I20260812 06:36:54.128597 10247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:54.130126 10011 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260812 06:36:54.130685 10011 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
W20260812 06:36:54.130837 10011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 78ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.131441  9611 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:54.131687  9885 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 06:36:54.131755  9885 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.131832  9885 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.132216  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
W20260812 06:36:54.132921  9387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:36:54.133050  9885 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.133869  9885 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.134192  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:54.134202  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.134308  8577 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.134325  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 1.
I20260812 06:36:54.134328  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.134716  9734 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.135437  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 2.
I20260812 06:36:54.135614  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:54.135865  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:54.136054  9510 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.136247  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:54.136358  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.136495  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.136616  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 1.
I20260812 06:36:54.137048  9387 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.137122  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 2.
I20260812 06:36:54.137305  9925 raft_consensus.cc:2764] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260812 06:36:54.137447  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 2.
I20260812 06:36:54.137639  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:54.137773  9272 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.137835  9611 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:54.138190  9885 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 06:36:54.138409  9885 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 LEADER]: Becoming Leader. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Running, Role: LEADER
I20260812 06:36:54.138746  9885 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.138900  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 2.
W20260812 06:36:54.140902  9964 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 20ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.140902 10147 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:54.142424  8072 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 reported cstate change: term changed from 1 to 2, leader changed from 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195) to c74db17d01e8469fadd613ce468bbc60 (127.7.212.199). New cstate: current_term: 2 leader_uuid: "c74db17d01e8469fadd613ce468bbc60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:54.140729 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:36:54.143692  9611 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:54.143925  9611 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.145766  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.145977  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.148164 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:36:54.148617  9387 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.151048  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.152309  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.152537  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.152750  8577 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.153168  9171 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a60c871d36ed4e6ea4b74231f1414739; no voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:54.153367 10248 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:54.153584 10248 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:54.154217 10248 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:54.154367  9892 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 06:36:54.154942  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.155085  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.155342 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:36:54.155725  9163 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.156276  8072 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:54.157265  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:58263
I20260812 06:36:54.196722  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:54.196949  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.198339  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.203742  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.203953  9272 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.204208  9734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.204546  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.204727  8577 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.205621 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:36:54.206625  8932 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.206772  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.207005  9879 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.207396  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.207557  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.208228  9879 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 06:36:54.208374  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.208570  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 2: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:54.209259 10011 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.108s	user 0.007s	sys 0.004s
I20260812 06:36:54.211614 10011 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.213387 10011 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:54.213907 10011 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.214566 10248 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:54.214828 10011 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
W20260812 06:36:54.219203 10272 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:54.219259 10272 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:54.220043  9044 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.220043  9272 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.220525 10272 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:54.220736  9885 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 1, Time since last communication: 0.000s
W20260812 06:36:54.222347  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 1ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.239327  9510 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.239883  9885 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 06:36:54.241539  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 18ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.244382  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.244923  9885 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 06:36:54.247283  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 135ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.252887 10144 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:36:54.252972  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.253549  9885 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:54.255370 10144 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 2 FOLLOWER]: Refusing update from remote peer c74db17d01e8469fadd613ce468bbc60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 06:36:54.256132  9885 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 06:36:54.264284 10291 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 33ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.283648  9204 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51684: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.283669  9207 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51684: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.287927  8477 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.288048  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.307173  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.307173  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.314949  8971 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:54.316572  8971 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:54.326804  9964 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 78ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.329118  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 63ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.329514 10291 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 23ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.330559  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 77ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.331195  9966 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 73ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.387203  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 37ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.394580  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 37ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.402138 10291 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 19ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.404377  9965 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 54ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.405732  9964 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 51ms of latency in SegmentAllocator::Sync()
E20260812 06:36:54.409423 10317 fault_injection.cc:59] injecting fault for kudu (pid 9583): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:54.429657  9995 connection.cc:573] server connection from 127.7.212.199:48147 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:54.430344  8454 connection.cc:573] server connection from 127.7.212.199:39251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:54.432332  8937 connection.cc:466] server connection from 127.7.212.199:59419 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.199:59419 (request call id 7) could send its response
W20260812 06:36:54.432916  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.435818  9204 tablet_service.cc:3080] error setting up scanner b07f034b6d4249f29b08f4135a67bfb3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.166s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:54.436319  9207 tablet_service.cc:3080] error setting up scanner 2e7f5efe2ddb464ebf197d0d5a3acca5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.167s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
W20260812 06:36:54.437302  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.438238  8019 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 06:36:54.438613  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.199:34285
--local_ip_for_outbound_sockets=127.7.212.199
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=34823
--webserver_interface=127.7.212.199
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:54.446892 10081 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.446910 10080 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.450205  9437 tablet_service.cc:3080] error setting up scanner 428488f6fd934f2bb367be17c61afb6d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.184s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
W20260812 06:36:54.451092  9438 tablet_service.cc:3080] error setting up scanner 1ef08898006b44d5acfe0a63896b3785 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.185s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:54.453727  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.458879  9172 connection.cc:466] server connection from 127.7.212.199:55211 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.199:55211 (request call id 9) could send its response
W20260812 06:36:54.463321  9207 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51684: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.469938  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.474582  9438 tablet_service.cc:3080] error setting up scanner daad3568adba4b4ba984e9b11ecb6914 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.208s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
W20260812 06:36:54.479550  8971 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:54.491222  9438 tablet_service.cc:3080] error setting up scanner 0d9e9e9715da4f3482049e5d9302e655 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.225s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:54.492919  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.517243  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.531960  9207 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51684: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.546792 10080 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.549350  9438 tablet_service.cc:3080] error setting up scanner 012bf7c5edb24ead82a914db17ddd91e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.283s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:54.552361  9438 tablet_service.cc:3080] error setting up scanner 9b99ac4234db40c0a63077fec579512d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.286s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
W20260812 06:36:54.561100 10359 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:54.561352 10359 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:54.561404 10359 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:54.561447 10359 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:54.561523 10359 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:54.561565 10359 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:54.561654 10359 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:54.562004  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.565241 10359 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:54.565428 10359 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.199
I20260812 06:36:54.569399 10359 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.199:34285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.7.212.199
--webserver_port=34823
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.10359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.199

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:54.570487 10359 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:54.571642 10359 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:54.574635 10359 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:54.577729  8971 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:54.578775 10370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:54.578783 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:54.578853 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:54.579166 10359 server_base.cc:1061] running on GCE node
I20260812 06:36:54.579533 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:54.580149 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:54.581359 10359 hybrid_clock.cc:648] HybridClock initialized: now 1786516614581339 us; error 36 us; skew 500 ppm
I20260812 06:36:54.583285 10359 webserver.cc:533] Webserver started at http://127.7.212.199:34823/ using document root <none> and password file <none>
I20260812 06:36:54.583877 10359 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:54.583995 10359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:54.587565 10359 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:54.589754 10380 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:54.590884 10359 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:36:54.591051 10359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:54.591512 10359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:54.594691  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.622035 10359 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:54.622789 10359 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:54.623003 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:54.623637 10359 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:54.625061 10389 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:54.628495  9925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.628659  9925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.629510  9925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.629637 10359 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:54.629695 10359 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
I20260812 06:36:54.629777 10359 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:54.629908  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.630544 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:54.630990  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.631368  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:54.631392  9391 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Network error: Client connection negotiation failed: client connection to 127.7.212.199:34285: connect: Connection refused (error 111)
I20260812 06:36:54.632012 10359 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:54.632076 10359 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
W20260812 06:36:54.632133  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.632433 10389 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap starting.
I20260812 06:36:54.632836  9387 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.633167  9925 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 06:36:54.633154  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.642895 10389 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:54.645291 10359 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.199:34285
I20260812 06:36:54.645375 10582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.199:34285 every 8 connection(s)
I20260812 06:36:54.646517 10359 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 06:36:54.647387  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 10359
W20260812 06:36:54.651885  9207 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51684: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.659102 10389 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=12 ignored=0} inserts{seen=10 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
I20260812 06:36:54.659684 10389 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap complete.
W20260812 06:36:54.659888 10389 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 49ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.660405 10583 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:54.660702 10583 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:54.661353 10583 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:54.663373  8072 ts_manager.cc:194] Re-registered known tserver with Master: c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:54.664227  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.199:53407
W20260812 06:36:54.671279  9438 tablet_service.cc:3080] error setting up scanner 4cf3f58767f544d4a6fb544478836cfc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.405s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:54.675915 10080 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.685171  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.685338  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.686156  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.686729  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.687470  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.687489  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.687646  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 2.
I20260812 06:36:54.687907  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:54.690339  9438 tablet_service.cc:3080] error setting up scanner 991d6c10a7c447cca134201b5dce0382 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.424s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
I20260812 06:36:54.691428 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:54.691885  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739
I20260812 06:36:54.692121  9949 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 06:36:54.693264 10487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.694847  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.694993  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:54.695170  8455 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.695833  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.695883  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.696009  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 2.
I20260812 06:36:54.696019  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.696383 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:54.696468  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.696550  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.696619  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 2.
W20260812 06:36:54.700038  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.700274 10487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
W20260812 06:36:54.700744  8936 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.700845  8936 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.700997  9879 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 06:36:54.701961 10290 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.702097 10290 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.703373 10290 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.704975  9892 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.705116  9892 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.705844  9892 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:54.707090  9884 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.707204  9884 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.707926  9884 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:54.707935  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.708055  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 2.
I20260812 06:36:54.708536  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.708655  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 2.
I20260812 06:36:54.708813  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.708917  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 2.
I20260812 06:36:54.708902  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.709043  9044 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 2.
I20260812 06:36:54.709864  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.709882  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.709987  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 2.
I20260812 06:36:54.709992  9272 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 2.
I20260812 06:36:54.710433 10389 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent bootstrapping tablet: real 0.078s	user 0.026s	sys 0.000s
I20260812 06:36:54.711547 10487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
W20260812 06:36:54.712100 10003 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:54.712298 10389 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.713403  9163 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:54.713497 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:54.713639 10144 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 2.
I20260812 06:36:54.713869  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.714002  9044 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 2.
I20260812 06:36:54.714289  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.714339  8214 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: 
I20260812 06:36:54.714349  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:54.714421  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 2.
I20260812 06:36:54.714529  9272 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 for term 3 because replica has last-logged OpId of term: 2 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 20.
I20260812 06:36:54.714555  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:54.714664  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 2.
I20260812 06:36:54.714859  9884 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 06:36:54.714911  9884 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.714954  9884 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.715001  9892 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 06:36:54.715049  9892 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c74db17d01e8469fadd613ce468bbc60)
I20260812 06:36:54.715093  9892 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.715394  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:54.715517  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 2.
I20260812 06:36:54.716117  9884 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.716117  9892 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.716979  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:54.717089  8577 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.717521 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:54.717655 10144 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.718195  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 3.
I20260812 06:36:54.718276  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:54.718406  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.718820 10144 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 3.
I20260812 06:36:54.719033  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:54.719169  8577 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 3: Already voted for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in this term.
I20260812 06:36:54.719571  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 3.
I20260812 06:36:54.719609  9892 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 3 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
W20260812 06:36:54.722709  8971 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:54.723664  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:54.723812  9043 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.723891 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:54.723886 10487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.724043  9042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:54.724578  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:54.724789  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 3: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:54.724853 10147 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:54.725104 10147 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 3: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:54.725081  9044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:54.725152 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:54.725303 10144 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 3: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:54.725309  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 3.
I20260812 06:36:54.726086  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:54.726104  9272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:54.726250  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 3: Already voted for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in this term.
I20260812 06:36:54.726285  9272 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 3: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:54.726907  9884 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 3 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:54.727159  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:54.727305  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 3: Already voted for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in this term.
I20260812 06:36:54.727665  9995 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:54.728123 10487 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:54.728193 10290 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 06:36:54.728462 10484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:54.729141  8217 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:54.729264 10389 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Initialized, Role: FOLLOWER
I20260812 06:36:54.729367  9884 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260812 06:36:54.729807 10389 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:54.730044 10485 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:36:54.730851 10389 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260812 06:36:54.731294 10583 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:54.732138 10485 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 3.
I20260812 06:36:54.733393  9171 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:54.733675  9892 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 06:36:54.733915  9892 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 LEADER]: Becoming Leader. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Running, Role: LEADER
I20260812 06:36:54.734438  9892 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.22, Last appended by leader: 22, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:54.735916  9965 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 1ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.737229  8072 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 reported cstate change: term changed from 2 to 3, leader changed from c74db17d01e8469fadd613ce468bbc60 (127.7.212.199) to a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197). New cstate: current_term: 3 leader_uuid: "a60c871d36ed4e6ea4b74231f1414739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:54.748291  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.774891  9892 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:54.774973  9892 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:54.775800  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.775964 10144 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.776021  9510 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.776261  9043 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.776471  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.776474 10627 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 13, Time since last communication: 0.000s
I20260812 06:36:54.776679 10621 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 06:36:54.777034 10485 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20260812 06:36:54.777042  9892 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 06:36:54.777156 10623 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 06:36:54.778126 10627 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 06:36:54.778483 10627 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 06:36:54.781564  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 56ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.783226  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 38ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.785616 10628 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 93ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.785727  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 77ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.825874 10287 mvcc.cc:204] Tried to move back new op lower bound from 7317572054115274752 to 7317572053956784128. Current Snapshot: MvccSnapshot[applied={T|T < 7317572052280020992}]
I20260812 06:36:54.826493  9954 mvcc.cc:204] Tried to move back new op lower bound from 7317572054115274752 to 7317572053956784128. Current Snapshot: MvccSnapshot[applied={T|T < 7317572052561313792}]
W20260812 06:36:54.828610  9965 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 20ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.829310  9966 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 93ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.833276 10291 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 20ms of latency in SegmentAllocator::Sync()
I20260812 06:36:54.842213  9959 mvcc.cc:204] Tried to move back new op lower bound from 7317572054115274752 to 7317572053956784128. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054115274752 or (T in {7317572054115274752})}]
W20260812 06:36:54.843452  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 48ms of latency in SegmentAllocator::Sync()
W20260812 06:36:54.843690  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 73ms of latency in SegmentAllocator::Sync()
E20260812 06:36:54.854275 10623 fault_injection.cc:59] injecting fault for kudu (pid 9134): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:54.875533  8936 connection.cc:573] server connection from 127.7.212.197:58429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:54.875852  9818 connection.cc:573] client connection to 127.7.212.197:41571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:54.876032  9818 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:54.876097  9947 meta_cache.cc:1510] marking tablet server a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571) as failed
W20260812 06:36:54.878607  8477 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.879348 10384 connection.cc:466] server connection from 127.7.212.197:40855 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:40855 (request call id 3) could send its response
W20260812 06:36:54.884718 10080 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.884738 10081 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:54.886516  8019 ts_itest-base.cc:575] TS 4 appears to have crashed. Restarting.
I20260812 06:36:54.886937  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.197:41571
--local_ip_for_outbound_sockets=127.7.212.197
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=33771
--webserver_interface=127.7.212.197
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:54.887023 10081 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46792: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:54.892001  8453 connection.cc:466] server connection from 127.7.212.197:44497 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:44497 (request call id 5) could send its response
W20260812 06:36:54.917251  9387 connection.cc:466] server connection from 127.7.212.197:33161 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:33161 (request call id 10) could send its response
W20260812 06:36:54.922940  8223 connection.cc:466] server connection from 127.7.212.197:38655 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:38655 (request call id 6) could send its response
W20260812 06:36:55.003786 10651 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:55.004012 10651 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:55.004031 10651 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:55.004052 10651 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:55.004138 10651 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:55.004187 10651 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:55.004325 10651 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:55.007788 10651 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:55.007910 10651 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.197
I20260812 06:36:55.011948 10651 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.197:41571
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.7.212.197
--webserver_port=33771
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.10651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.197

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:55.013154 10651 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:55.014329 10651 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:55.017145 10651 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:55.021294 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:55.021302 10657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:55.021302 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:55.021585 10651 server_base.cc:1061] running on GCE node
I20260812 06:36:55.022019 10651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:55.022635 10651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:55.023859 10651 hybrid_clock.cc:648] HybridClock initialized: now 1786516615023849 us; error 34 us; skew 500 ppm
I20260812 06:36:55.025887 10651 webserver.cc:533] Webserver started at http://127.7.212.197:33771/ using document root <none> and password file <none>
I20260812 06:36:55.026587 10651 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:55.026762 10651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:55.030511 10651 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.004s	sys 0.000s
I20260812 06:36:55.032681 10675 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:55.033862 10651 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:55.034003 10651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:55.034449 10651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:55.046105 10651 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:55.046723 10651 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:55.046921 10651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:55.047474 10651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:55.048652 10690 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:55.051966 10651 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:55.052044 10651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:55.052140 10651 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:55.053666 10651 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:55.053735 10651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:55.054080 10690 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap starting.
I20260812 06:36:55.062415 10690 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:55.064947 10651 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.197:41571
I20260812 06:36:55.064971 10874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.197:41571 every 8 connection(s)
I20260812 06:36:55.065903 10651 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 06:36:55.074445 10875 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:55.074677 10875 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:55.074712  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 10651
I20260812 06:36:55.075201 10875 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:55.077023  8074 ts_manager.cc:194] Re-registered known tserver with Master: a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571)
I20260812 06:36:55.077811  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.197:60811
I20260812 06:36:55.080983 10690 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
I20260812 06:36:55.081406 10690 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap complete.
W20260812 06:36:55.081575 10690 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 35ms of latency in SegmentAllocator::Sync()
I20260812 06:36:55.082389 10614 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.082559 10614 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.084241 10614 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.084467  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.084589  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.085366  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.086596 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.086973  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.087177  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.087425 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.087564 10485 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 3.
I20260812 06:36:55.087837  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.088428  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, c74db17d01e8469fadd613ce468bbc60; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.088660  9879 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 06:36:55.091117 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.092830  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.094674  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.095031  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.094537 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.095697 10382 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.096781 10614 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 06:36:55.097318 10382 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:55.097935  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.098069  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 3.
I20260812 06:36:55.098529 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:36:55.098965  8932 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:55.117208 10690 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent bootstrapping tablet: real 0.063s	user 0.020s	sys 0.008s
I20260812 06:36:55.118992 10690 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.121639 10690 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Initialized, Role: FOLLOWER
I20260812 06:36:55.122148 10690 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 3.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.122989 10875 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:55.123173 10690 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260812 06:36:55.128945  9884 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.129062  9884 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.129827  9884 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:55.130180  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.130651 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.130719 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.130719  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.130869 10485 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 3.
I20260812 06:36:55.130869  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 3.
I20260812 06:36:55.130992  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.133296 10290 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.133402 10290 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.134230 10290 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.134706 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.134888  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.134886 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.134941 10798 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 for term 4 because replica has last-logged OpId of term: 3 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 25.
I20260812 06:36:55.135013 10485 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 3.
I20260812 06:36:55.135259  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.135401  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 3.
I20260812 06:36:55.135713  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.136152  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.136286  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 3.
I20260812 06:36:55.136495  8214 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.136622  9995 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.136751  9884 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 06:36:55.136974 10290 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 06:36:55.137043 10290 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.137094 10290 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.138036 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.138171 10798 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 for term 4 because replica has last-logged OpId of term: 3 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 27.
I20260812 06:36:55.138159 10290 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.138839 10290 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 4 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.139009  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:55.139243  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:55.139315 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:55.139438 10485 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.139941  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:55.140010 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:55.140059  9043 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.140117 10798 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.140583 10485 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 4.
I20260812 06:36:55.140846  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:55.140954  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.141058  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 4.
I20260812 06:36:55.141458  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.141557  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.142050  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 4.
I20260812 06:36:55.142131  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.142199 10798 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 for term 4 because replica has last-logged OpId of term: 3 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 27.
I20260812 06:36:55.142598 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.142759 10485 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.142773  9995 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.143088 10290 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 06:36:55.143219  9925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:55.143313 10290 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:36:55.143337  9925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.143646 10290 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.143922  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.144071  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.144745 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.144723  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.144949  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 3.
I20260812 06:36:55.145258  9925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.146972  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.147113  8330 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.148104  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 3 to 4, leader changed from a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197) to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 4 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:55.150193  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.150333  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.150362 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.150522 10798 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 for term 4 because replica has last-logged OpId of term: 3 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 33.
I20260812 06:36:55.151548 10144 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.143229  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.151947  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:55.152081  9510 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 for term 4 because replica has last-logged OpId of term: 3 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 33.
I20260812 06:36:55.152137  9949 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.152523  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.152678  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.153190  9949 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 06:36:55.155402 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.155637 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 4.
I20260812 06:36:55.156863 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.157018 10485 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 4: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.157294  9391 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:55.157575  9925 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 3 FOLLOWER]: Advancing to term 4
I20260812 06:36:55.158665  9925 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260812 06:36:55.222566 10894 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:55.222669 10894 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:55.223662  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 4 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 28. (term mismatch)
I20260812 06:36:55.223827  8577 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting all ops after (but not including) 27
I20260812 06:36:55.223946  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.28
I20260812 06:36:55.224423  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.29
I20260812 06:36:55.224663  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.30
I20260812 06:36:55.224963  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.31
I20260812 06:36:55.225265  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.32
I20260812 06:36:55.225507  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 3.33
I20260812 06:36:55.226296 10894 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:55.227607  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 109ms of latency in SegmentAllocator::Sync()
I20260812 06:36:55.240602  9510 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 4 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 4 index: 28. (term mismatch)
I20260812 06:36:55.240690  9510 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting all ops after (but not including) 27
I20260812 06:36:55.240832  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.28
I20260812 06:36:55.241254  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.29
I20260812 06:36:55.241526  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.30
I20260812 06:36:55.241801  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.31
I20260812 06:36:55.242050  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.32
I20260812 06:36:55.242290  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.33
I20260812 06:36:55.242525  9510 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Aborting uncommitted WRITE_OP operation due to leader change: 3.34
I20260812 06:36:55.243302 10894 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:55.244616  9967 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 103ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.245702  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.247241  9043 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 24. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20260812 06:36:55.247788 10894 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 06:36:55.248148  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.248200  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.248205  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.249826 10485 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 24. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
W20260812 06:36:55.249838  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.250094  9957 mvcc.cc:204] Tried to move back new op lower bound from 7317572054115274752 to 7317572053956784128. Current Snapshot: MvccSnapshot[applied={T|T < 7317572052283244544}]
I20260812 06:36:55.250903 10894 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 06:36:55.252141  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.252496  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 22ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.253795  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.254077 10613 mvcc.cc:204] Tried to move back new op lower bound from 7317572054115274752 to 7317572053956784128. Current Snapshot: MvccSnapshot[applied={T|T < 7317572052283244544}]
W20260812 06:36:55.255398  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.256639 10628 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 25ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.257318  8477 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.257292  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.257318  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.260228  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.261412  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.262909  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.264277 10798 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 4 index: 28. (term mismatch)
I20260812 06:36:55.264353 10798 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting all ops after (but not including) 27
I20260812 06:36:55.264427 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.28
W20260812 06:36:55.264565  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.265012 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.29
I20260812 06:36:55.265247 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.30
I20260812 06:36:55.265434 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.31
I20260812 06:36:55.265625 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.32
I20260812 06:36:55.265802 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.33
W20260812 06:36:55.265789  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.265895  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.265988 10798 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Aborting uncommitted WRITE_OP operation due to leader change: 3.34
I20260812 06:36:55.266817 10894 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 24, Time since last communication: 0.000s
E20260812 06:36:55.267007 10894 fault_injection.cc:59] injecting fault for kudu (pid 9968): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:55.269053  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.270428  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.271601  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.278553 10422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.278566 10423 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.280232  8937 connection.cc:573] server connection from 127.7.212.195:41933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:55.280719  8019 ts_itest-base.cc:575] TS 2 appears to have crashed. Restarting.
W20260812 06:36:55.281018  9818 connection.cc:573] client connection to 127.7.212.195:40613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:55.281149  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:55.282083 10384 connection.cc:466] server connection from 127.7.212.195:36107 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:36107 (request call id 4) could send its response
W20260812 06:36:55.283564 10423 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.283751 10422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.283864 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.290596  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.290649  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.292910  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.293128  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.294428  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.296623  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.297801  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.307715 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.307713 10731 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.307713 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.308414 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.309670 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.311834 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.312747 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.323499 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.329154  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.336335 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.337090  8453 connection.cc:466] server connection from 127.7.212.195:47115 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:47115 (request call id 4) could send its response
W20260812 06:36:55.337574 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.338582  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.341519  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.345993  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.348078  9397 connection.cc:466] server connection from 127.7.212.195:59977 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:59977 (request call id 4) could send its response
W20260812 06:36:55.351392  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.353842 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.356271 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.356271 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.359007 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.364395  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.364446  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.368119  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.371423  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.371435  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.372521  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.375626  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.379149 10422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.379184 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.383388 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.389797  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.389894  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.389910  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.393220  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.394411  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.395787  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.398995  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.406273  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.406445  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.408887 10917 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:55.409063 10917 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:55.409081 10917 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:55.409096 10917 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:55.409132 10917 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:55.409150 10917 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:55.409191 10917 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:55.410620  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.411666  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.411710  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.412958 10917 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:55.413100 10917 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
W20260812 06:36:55.416422  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.417598 10917 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.10917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:55.418766 10917 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:55.419982 10917 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:55.422959  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
E20260812 06:36:55.422987 10917 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:55.422999  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.426663 10930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:55.426674 10926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:55.426782 10927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:55.427313 10917 server_base.cc:1061] running on GCE node
I20260812 06:36:55.427681 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:55.428220 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:55.429407 10917 hybrid_clock.cc:648] HybridClock initialized: now 1786516615429387 us; error 35 us; skew 500 ppm
W20260812 06:36:55.430371 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.430387 10422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.431375 10917 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:55.431941 10917 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:55.431999 10917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 06:36:55.433575  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.435999 10917 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:55.438149 10941 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:55.439428 10917 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:55.439531 10917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:55.439944 10917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:55.442435 10421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.442456 10422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42188: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.456319  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.456518 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.457485 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.467628 10917 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:55.468326 10917 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:55.468518 10917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:55.469198 10917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:55.470444 10949 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:55.474023 10917 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
W20260812 06:36:55.474001  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.474105 10917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:55.474180 10917 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:55.475826 10917 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:55.475896 10917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:36:55.476254 10949 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
W20260812 06:36:55.478006  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.479835 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.484329 10949 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:55.487990 10917 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
W20260812 06:36:55.488734  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.488802 11120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:55.489162 10917 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:55.496634 11121 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:55.496893 11121 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:55.497390 11121 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:55.499235  8074 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:55.499279  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 10917
I20260812 06:36:55.500221  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:49891
I20260812 06:36:55.504528 10949 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20260812 06:36:55.505003 10949 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
I20260812 06:36:55.505302 10949 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.029s	user 0.028s	sys 0.000s
I20260812 06:36:55.507287 10949 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.509347 10949 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:55.509846 10949 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 4.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.510550 11121 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:55.510771 10949 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
W20260812 06:36:55.511333 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.520944  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.521919  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.525290 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.526389 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.528728  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.528818  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.533035  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.544296  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.544405  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.548667  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.549424  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:55.549589  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.550354  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.550581  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.551159 10485 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.551142 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.551182  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.551298 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 4.
I20260812 06:36:55.551329  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 4.
W20260812 06:36:55.551757  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.551852  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.552497  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.553226 10915 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:55.553369 10915 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.554059 10915 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.554282  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.554775  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.554909  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 4.
I20260812 06:36:55.554993  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.555125  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 4.
I20260812 06:36:55.556003 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.555997 11043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.556129 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 4.
I20260812 06:36:55.556305 11043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 4.
W20260812 06:36:55.556362  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.556445 10382 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:55.556618 10915 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 06:36:55.556674 10915 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:55.556740 10915 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.557721 10915 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.558367  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739; no voters: 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.558547 10915 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 5 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.558894 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:55.559013 10798 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.559015  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:55.559144  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:55.559154 11041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:55.559257  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.559273 11041 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.559338 11043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.560093 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 5.
I20260812 06:36:55.560338  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:55.560344  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 5.
I20260812 06:36:55.560425  9043 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.561513 11041 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 5.
I20260812 06:36:55.561501  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 5.
I20260812 06:36:55.561687  9879 raft_consensus.cc:2764] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260812 06:36:55.561910 10382 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:55.562024  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.562075  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:55.562175 10915 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 06:36:55.562261 10915 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 LEADER]: Becoming Leader. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Running, Role: LEADER
W20260812 06:36:55.562599  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.562587 10915 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.565250  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 reported cstate change: term changed from 4 to 5, leader changed from 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195) to c74db17d01e8469fadd613ce468bbc60 (127.7.212.199). New cstate: current_term: 5 leader_uuid: "c74db17d01e8469fadd613ce468bbc60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:55.568737  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.569674  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.572667  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.576928  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.578083 10628 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 19ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.578146 10915 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:55.578197 10915 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 06:36:55.578509 10614 fault_injection.cc:59] injecting fault for kudu (pid 10359): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:55.597952  9823 connection.cc:573] client connection to 127.7.212.199:34285 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:55.599232  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.602049  8019 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 06:36:55.602500  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.199:34285
--local_ip_for_outbound_sockets=127.7.212.199
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=34823
--webserver_interface=127.7.212.199
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:55.605571  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.622267  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.623623  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.623638  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.626127  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.626271  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.628690  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:55.628877  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.629745  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.629979 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.630198 10798 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:55.630578  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.630786  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:55.630908  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.631233  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.631392  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
W20260812 06:36:55.632408  8455 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Network error: Client connection negotiation failed: client connection to 127.7.212.199:34285: connect: Connection refused (error 111)
I20260812 06:36:55.632517  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.632859  9949 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.634152  9949 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 06:36:55.634848 11041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.635138 11041 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:55.645165  9925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:55.645344  9925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.646046  9925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.646869  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.646907  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.647032  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 5.
I20260812 06:36:55.647078  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:55.647323 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.647505 10798 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:36:55.647480  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.647630  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
W20260812 06:36:55.648551 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.648638  9391 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Network error: Client connection negotiation failed: client connection to 127.7.212.199:34285: connect: Connection refused (error 111)
I20260812 06:36:55.648725  9391 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:55.649101  9925 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 4 FOLLOWER]: Advancing to term 5
I20260812 06:36:55.650327  9925 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 06:36:55.651862 11041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.652040 11041 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 5: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
W20260812 06:36:55.653290 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.655805  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.660213  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.662559  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.682628  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.682634  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.684016 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.689810 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.694067 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.711664 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.712647 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.713279 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.719977 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.726521 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.727228 11140 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:55.727423 11140 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:55.727442 11140 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:55.727458 11140 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:55.727494 11140 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:55.727519 11140 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:55.727560 11140 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:55.730908 11140 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:55.731005 11140 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.199
I20260812 06:36:55.734862 11140 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.199:34285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.7.212.199
--webserver_port=34823
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.11140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.199

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:55.735972 11140 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:55.737169 11140 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:55.740012 11140 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:55.742156 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.743921 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:55.743965 11152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:55.744235 11140 server_base.cc:1061] running on GCE node
W20260812 06:36:55.744311 10954 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.744424 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.744505 11161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:55.745471 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260812 06:36:55.745606 10953 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.745977 10954 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46892: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.746258 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:55.747448 11140 hybrid_clock.cc:648] HybridClock initialized: now 1786516615747427 us; error 41 us; skew 500 ppm
I20260812 06:36:55.749387 11140 webserver.cc:533] Webserver started at http://127.7.212.199:34823/ using document root <none> and password file <none>
I20260812 06:36:55.750106 11140 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:55.750170 11140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 06:36:55.750473  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.754180 11140 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:55.756594 11171 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 06:36:55.757073  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.757838 11140 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:36:55.757961 11140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:55.758368 11140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:55.774770  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.775125  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.775187  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.776734 11140 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:55.777524 11140 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20260812 06:36:55.777513  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.777731 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:55.778450 11140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 06:36:55.778669  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.780164 11181 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:55.783849 11140 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:55.783946 11140 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:55.784031 11140 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 06:36:55.785392  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.785682 11140 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:55.785792 11140 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.001s
I20260812 06:36:55.786103 11181 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap starting.
W20260812 06:36:55.791899  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.796473 11181 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:55.797907 11140 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.199:34285
I20260812 06:36:55.798043 11393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.199:34285 every 8 connection(s)
I20260812 06:36:55.798909 11140 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 06:36:55.800886  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 11140
W20260812 06:36:55.806725  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.807893  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.808032 11394 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:55.808248 11394 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:55.808914 11394 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
W20260812 06:36:55.808957  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.810012  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:55.810124  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:55.810772  8077 ts_manager.cc:194] Re-registered known tserver with Master: c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:55.811810  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.199:51717
I20260812 06:36:55.817116 11181 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 6 replicates
I20260812 06:36:55.817593 11181 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap complete.
W20260812 06:36:55.817596  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.817778 11181 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 156ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.826201  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.838831  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.842275  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.842376  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.843636  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.845963  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.895010  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.911047  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.911562  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.914752  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.915998 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.922847 11127 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:55.923025 11127 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.924683 11127 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.930596  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.930755  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 5.
I20260812 06:36:55.930995 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.931136 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 5.
I20260812 06:36:55.932862  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:55.932972  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.931969 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.933626  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.933670  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:55.933815  9043 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 5.
W20260812 06:36:55.934423 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.934587  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.934671  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.934743  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 5.
I20260812 06:36:55.934793  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 5.
I20260812 06:36:55.934813  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:55.934952  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 5.
I20260812 06:36:55.935119 10942 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: 
I20260812 06:36:55.935269 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:55.935388 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 5.
I20260812 06:36:55.935406 11127 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260812 06:36:55.935384 10944 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 pre-election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:55.935472 11127 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:55.935511 11127 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.935604 11041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:55.935631  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:55.935772  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 5.
W20260812 06:36:55.936064 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:55.936652 11127 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.936710  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:55.936852  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 5.
I20260812 06:36:55.936846  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:55.937026  9879 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 06:36:55.937071  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:55.937104  9879 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.937346 11127 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.937546 11041 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 6: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.938112  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:55.938156  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.938457  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:55.938576  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.938736  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:55.938853  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.938882  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:55.939073  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 6: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:55.939040  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:55.939175  8577 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.939815  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 6.
I20260812 06:36:55.939836  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:55.939949  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 6: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.939929  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 6.
I20260812 06:36:55.940233 10797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:55.940268  8577 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 6.
I20260812 06:36:55.940378 10797 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 5 FOLLOWER]: Advancing to term 6
I20260812 06:36:55.940871 11041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:55.941005 11041 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 6: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.941020 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:55.940740  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:55.941390  8577 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 6: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.941498 10797 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 6.
W20260812 06:36:55.941547 10944 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:55.941781  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:55.941829 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:55.941867 10942 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:55.941929  9511 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 6: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:55.941990 10798 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 6: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:55.942070 11127 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 06:36:55.942248  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, a60c871d36ed4e6ea4b74231f1414739; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:55.942286 11127 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:36:55.942480  9879 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260812 06:36:55.942674 11127 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 4.28, Last appended by leader: 28, Current term: 6, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.943486 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:55.943573 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 6 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
W20260812 06:36:55.943924  8936 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:55.944237  8936 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 6 pre-election: Tablet error from VoteRequest() call to peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:55.946622  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 5 to 6, leader changed from c74db17d01e8469fadd613ce468bbc60 (127.7.212.199) to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 6 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:55.948000  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.950626 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:55.952867 11419 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 42ms of latency in SegmentAllocator::Sync()
W20260812 06:36:55.952913 11127 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:55.953085 11127 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:55.954310  9043 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20260812 06:36:55.954444 10798 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20260812 06:36:55.954608  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
W20260812 06:36:55.954674 10944 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 -> Peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Couldn't send request to peer c74db17d01e8469fadd613ce468bbc60. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 06:36:55.954761  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 3 index: 25. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20260812 06:36:55.955224  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20260812 06:36:55.955204 11424 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:55.955929 11127 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.001s
I20260812 06:36:55.956969 11418 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:55.958701  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 57ms of latency in SegmentAllocator::Sync()
I20260812 06:36:55.960593 11423 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:55.960848 11420 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
E20260812 06:36:55.961484 11424 fault_injection.cc:59] injecting fault for kudu (pid 10917): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 06:36:55.975531 11181 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent bootstrapping tablet: real 0.190s	user 0.028s	sys 0.004s
I20260812 06:36:55.977644 11181 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Replica starting. Triggering 6 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.980001 11181 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Initialized, Role: FOLLOWER
I20260812 06:36:55.980705 11181 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 5.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:55.981729 11181 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260812 06:36:55.981845 11394 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
W20260812 06:36:55.988468  8460 connection.cc:573] server connection from 127.7.212.195:34935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:55.988749  8215 connection.cc:466] server connection from 127.7.212.195:48073 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:48073 (request call id 3) could send its response
W20260812 06:36:55.989056  9818 connection.cc:573] client connection to 127.7.212.195:40613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:55.989161  9818 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:55.990362  8019 ts_itest-base.cc:575] TS 2 appears to have crashed. Restarting.
I20260812 06:36:55.990700  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:56.016448  8937 connection.cc:466] server connection from 127.7.212.195:50069 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:50069 (request call id 3) could send its response
W20260812 06:36:56.027397  9436 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.027472  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.027397  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.028111  9435 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.028895  9435 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.028961  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.028961  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.066052  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.068439  8970 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.068591  8968 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.069103  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.070190  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.109334  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.109367  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.110240  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.110669  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.110960  8969 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50966: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.119657 11437 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:56.119889 11437 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:56.119916 11437 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:56.119966 11437 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:56.120052 11437 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:56.120102 11437 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:56.120222 11437 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:56.124011 11437 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:56.124154 11437 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:36:56.128954 11437 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.11437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:56.130398 11437 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:56.131649 11437 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:56.134891 11437 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:56.138980 11445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:56.139027 11443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:56.139299 11437 server_base.cc:1061] running on GCE node
W20260812 06:36:56.138986 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:56.139865 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:56.140394 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:56.141569 11437 hybrid_clock.cc:648] HybridClock initialized: now 1786516616141557 us; error 32 us; skew 500 ppm
I20260812 06:36:56.143438 11437 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:56.144073 11437 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:56.144137 11437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:56.148452 11437 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:36:56.153064 11463 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 06:36:56.154287 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.154292  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.154376 11437 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.002s
W20260812 06:36:56.154822 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.154466 11437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:56.154963 11437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:56.181532 11437 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:56.182189 11437 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:56.182348 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:56.182901 11437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:56.184139 11471 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:56.187855 11437 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:56.187933 11437 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:56.187994 11437 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:56.189602 11437 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:56.189656 11437 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:56.189985 11471 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
W20260812 06:36:56.195385 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.195602 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.197245  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.197909 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.200475 11471 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:56.201965 11437 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:36:56.201982 11661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:56.202849 11437 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:56.208703  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 11437
I20260812 06:36:56.211714 11664 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:56.211954 11664 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:56.212445 11664 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:56.214144  8077 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:56.215108  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:36651
I20260812 06:36:56.220116 11471 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 6 replicates
I20260812 06:36:56.220537 11471 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
I20260812 06:36:56.220898 11471 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.031s	user 0.018s	sys 0.011s
I20260812 06:36:56.222829 11471 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Replica starting. Triggering 6 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.224495 11471 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:56.224979 11471 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.225625 11664 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:56.225711 11471 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W20260812 06:36:56.239853  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.245829 11510 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.245829 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.245829 11507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.245885 11509 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.258451  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:56.258596  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.259361  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.259773  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.259851 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.259889 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.260056 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 6.
I20260812 06:36:56.260123 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 5.
I20260812 06:36:56.260530 11428 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:56.260551  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.260704  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 6.
I20260812 06:36:56.260666 11428 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.260753  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.260939  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 6.
I20260812 06:36:56.261415  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:56.261442 11428 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:56.261806 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.261947 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 6.
I20260812 06:36:56.262300  9879 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 06:36:56.262367  9879 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:56.262414  9879 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.262552  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.263010  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.263137  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 6.
I20260812 06:36:56.263581  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:56.263723  9879 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.263975  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.265067  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:56.265187 11570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.265503 11570 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 6.
I20260812 06:36:56.265530 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:56.265633 10798 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.266630 11570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:56.266741 11570 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.266741 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 7.
I20260812 06:36:56.267802  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:56.267925  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.268728 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.268893 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 5.
I20260812 06:36:56.269145  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 7.
I20260812 06:36:56.269264  8217 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:56.269486 11428 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 06:36:56.269544 11428 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:56.269591 11428 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.269543 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.269819  9879 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 7 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.270241  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:56.270777 11428 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.271318 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:56.271579 11428 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 7 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:56.271682  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a in current term 7: Already voted for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in this term.
I20260812 06:36:56.271615 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 5 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.272364 11570 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 7.
I20260812 06:36:56.272681  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:56.272822  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.273037  8577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:56.273169  8577 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 6 FOLLOWER]: Advancing to term 7
I20260812 06:36:56.273200 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:56.273391 11572 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.273542  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.273831  8932 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:36:56.274629 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:56.275398  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.275560  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.276157  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:56.276152  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:56.276186 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 7.
I20260812 06:36:56.276304  9511 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.276304  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.276539 10797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.276697 10797 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.276697 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:56.276921 10798 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.276954  8214 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:56.277022  8577 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 for term 7 because replica has last-logged OpId of term: 6 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 30.
I20260812 06:36:56.277177 11428 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260812 06:36:56.277824  9043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:56.277863  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.277984  9043 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.278045  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in this term.
I20260812 06:36:56.278056  9879 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Leader election won for term 7
I20260812 06:36:56.278465  9879 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 LEADER]: Becoming Leader. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Running, Role: LEADER
I20260812 06:36:56.278450  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:56.278653  9949 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260812 06:36:56.278872  9879 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.30, Last appended by leader: 30, Current term: 7, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:56.280761  9958 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 06:36:56.281813 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.282475 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.282650 11572 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.282606 11306 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 7: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.282682  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a reported cstate change: term changed from 6 to 7, leader changed from 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195) to 1adfc3d9d367476090a9513b769ac76a (127.7.212.196). New cstate: current_term: 7 leader_uuid: "1adfc3d9d367476090a9513b769ac76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:56.283073 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.291680  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.293910  8512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.294039  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.296156  9879 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:56.296257  9879 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:56.297087  8577 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.297192 11306 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.297488 11572 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.297766  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.297729 11684 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.297859 11684 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Peer 9f32be8bb322423eaa290fd71a8baf10 log is divergent from this leader: its last log entry 6.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260812 06:36:56.297904  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.298066 11689 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.298249 11689 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Peer c74db17d01e8469fadd613ce468bbc60 log is divergent from this leader: its last log entry 5.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260812 06:36:56.298499 11686 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.298631 10798 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20260812 06:36:56.298678 11687 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.298684  8577 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 3.24->[3.25-7.32]   Dedup: 6.30->[7.31-7.32]
I20260812 06:36:56.298808  8577 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting all ops after (but not including) 30
I20260812 06:36:56.298834  8577 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 6.31
I20260812 06:36:56.298801  9879 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 24, Time since last communication: 0.000s
E20260812 06:36:56.299000  9879 fault_injection.cc:59] injecting fault for kudu (pid 8905): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:56.318457  8453 connection.cc:573] server connection from 127.7.212.196:40703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:56.318614  9944 meta_cache.cc:1510] marking tablet server 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991) as failed
I20260812 06:36:56.318848  9947 meta_cache.cc:1510] marking tablet server 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991) as failed
I20260812 06:36:56.319204  8019 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 06:36:56.319602  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.196:38991
--local_ip_for_outbound_sockets=127.7.212.196
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=40525
--webserver_interface=127.7.212.196
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:56.323841  9438 tablet_service.cc:3080] error setting up scanner 22eb11df13f947ad9a01455fe4d9cdf0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.079s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "22deccacb2c645f994ae5f0fb699ef3b"
W20260812 06:36:56.324184  8256 tablet_service.cc:3080] error setting up scanner 4ed6d73024bf4d3eb63d7dd3e9b928e5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.364s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572052280020992 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c434212af27d4c76a02c1ce9e78bf0e9"
W20260812 06:36:56.336476  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.338621  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.341739  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.348232  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.376907  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.385402 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.389999 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.390596 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.425364 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.433491 11691 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:56.433703 11691 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:56.433722 11691 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:56.433738 11691 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:56.433773 11691 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:56.433791 11691 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:56.433841 11691 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:56.435163  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.435254  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.436270  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.437939 11691 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:56.438102 11691 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.196
I20260812 06:36:56.442310 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.196:38991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.7.212.196
--webserver_port=40525
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.196

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:56.443506 11691 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:56.444741 11691 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:56.448078 11691 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:56.451907 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:56.451952 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:56.451911 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:56.452088 11691 server_base.cc:1061] running on GCE node
I20260812 06:36:56.452512 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:56.453135 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:56.454334 11691 hybrid_clock.cc:648] HybridClock initialized: now 1786516616454322 us; error 45 us; skew 500 ppm
I20260812 06:36:56.456204 11691 webserver.cc:533] Webserver started at http://127.7.212.196:40525/ using document root <none> and password file <none>
I20260812 06:36:56.456830 11691 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:56.456960 11691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:56.460316 11691 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:36:56.462461 11718 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:56.463565 11691 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:56.463730 11691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:56.464082 11691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:56.472854  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.479341 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.483708 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.498239 11691 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:56.498893 11691 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:56.499096 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:56.499691 11691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:56.500970 11730 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:56.504135 11691 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:56.504222 11691 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s	user 0.001s	sys 0.000s
I20260812 06:36:56.504329 11691 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:56.505906 11691 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:56.505978 11691 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:56.506273 11730 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap starting.
I20260812 06:36:56.512619 11730 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:56.517668 11691 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.196:38991
I20260812 06:36:56.517719 11905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.196:38991 every 8 connection(s)
I20260812 06:36:56.518550 11691 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
W20260812 06:36:56.524176 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.525943 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.527732  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 11691
I20260812 06:36:56.527892 11906 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:56.528132 11906 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:56.528609 11906 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:56.530294  8077 ts_manager.cc:194] Re-registered known tserver with Master: 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991)
I20260812 06:36:56.531052  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.196:51257
I20260812 06:36:56.531641 11730 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 7 replicates
I20260812 06:36:56.532140 11730 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap complete.
W20260812 06:36:56.532383 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.532506 11730 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent bootstrapping tablet: real 0.026s	user 0.018s	sys 0.005s
I20260812 06:36:56.534543 11730 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Replica starting. Triggering 7 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.536584 11730 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Initialized, Role: FOLLOWER
I20260812 06:36:56.537066 11730 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.537715 11906 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:56.538177 11730 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent starting tablet: real 0.006s	user 0.009s	sys 0.000s
W20260812 06:36:56.540979 11772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52204: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.573050 11772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52204: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.577662 11772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52204: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.585227 11772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52204: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.593743  8511 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51428: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.600397  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:56.600523  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.601331  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.601517 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.601672 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 7.
I20260812 06:36:56.602083 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.602221 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 7.
I20260812 06:36:56.602401  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.602468 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.602555  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 7.
I20260812 06:36:56.602598 11572 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 7.
I20260812 06:36:56.603142  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.603159  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 
I20260812 06:36:56.603302  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 7.
I20260812 06:36:56.603415  9949 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 06:36:56.603493  9949 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:56.603554  9949 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.604552  9949 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.605262  9949 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 8 election: Requested vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.605779  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:56.605794 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:56.605912  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.605912 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.605859 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:56.606007 11572 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.606920  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
I20260812 06:36:56.606920 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
I20260812 06:36:56.607216 11572 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
I20260812 06:36:56.607499 11840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:56.607738 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:56.607770 11840 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.607779  8453 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60; no voters: 
I20260812 06:36:56.607764  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:56.607872 10798 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.607895  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 7 FOLLOWER]: Advancing to term 8
I20260812 06:36:56.607959  9949 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 06:36:56.608127  9949 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 LEADER]: Becoming Leader. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Running, Role: LEADER
I20260812 06:36:56.608470  9949 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.32, Last appended by leader: 32, Current term: 8, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.608911 10798 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
I20260812 06:36:56.608995  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
I20260812 06:36:56.607524 11841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:56.610148 11840 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 8.
W20260812 06:36:56.610638  9962 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 162ms of latency in SegmentAllocator::Sync()
I20260812 06:36:56.611845  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 reported cstate change: term changed from 7 to 8, leader changed from 1adfc3d9d367476090a9513b769ac76a (127.7.212.196) to 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194). New cstate: current_term: 8 leader_uuid: "9f32be8bb322423eaa290fd71a8baf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:56.624148 11919 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:56.624269 11919 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:56.625138 11306 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
I20260812 06:36:56.625160  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
I20260812 06:36:56.625160  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
I20260812 06:36:56.625469 11572 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
I20260812 06:36:56.625742  9949 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.625917 11924 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.626289 11840 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 7 index: 31. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
I20260812 06:36:56.626217 11923 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.626745 10798 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 8 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 8 index: 34. (index mismatch)
W20260812 06:36:56.628373 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 51ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.628722 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 06:36:56.628957 11923 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Peer c74db17d01e8469fadd613ce468bbc60 log is divergent from this leader: its last log entry 5.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
W20260812 06:36:56.628963 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.629268  9949 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.629787 11921 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.629954 11924 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:56.630439 11306 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 3.24->[3.25-8.34]   Dedup: 4.28->[6.29-8.34]
I20260812 06:36:56.630605 11306 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Aborting all ops after (but not including) 28
I20260812 06:36:56.630669 11306 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Aborting uncommitted NO_OP operation due to leader change: 5.29
I20260812 06:36:56.630728 11306 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 06:36:56.630815 11306 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Aborting uncommitted WRITE_OP operation due to leader change: 5.30
W20260812 06:36:56.634044 11937 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 105ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.634924 11938 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 4ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.635327 11933 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 65ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.637996 11936 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 10ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.641225 11938 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 68ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.644119 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.680991  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.681521 11943 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51428 (ReqId={client: 4105b259dcfe4c81b5e5a4faec14f6a3, seq_no=21, attempt_no=2}) took 1798 ms. Trace:
E20260812 06:36:56.682147  9949 fault_injection.cc:59] injecting fault for kudu (pid 8428): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:56.689101 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 109ms of latency in SegmentAllocator::Sync()
W20260812 06:36:56.695426 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.696895 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.703770 11464 connection.cc:573] server connection from 127.7.212.194:39797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:56.704185  8019 ts_itest-base.cc:575] TS 1 appears to have crashed. Restarting.
I20260812 06:36:56.704602  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:34921
--local_ip_for_outbound_sockets=127.7.212.194
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=41195
--webserver_interface=127.7.212.194
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:56.709803 11174 connection.cc:466] server connection from 127.7.212.194:53217 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.194:53217 (request call id 5) could send its response
W20260812 06:36:56.740451 10683 connection.cc:466] server connection from 127.7.212.194:58571 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.194:58571 (request call id 6) could send its response
W20260812 06:36:56.744374 11772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52204: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:56.750509  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.792188  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.797125 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.798633  8215 connection.cc:466] server connection from 127.7.212.194:38843 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.194:38843 (request call id 9) could send its response
W20260812 06:36:56.804108 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.821825 11962 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:56.822005 11962 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:56.822023 11962 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:56.822038 11962 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:56.822151 11962 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:56.822198 11962 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:56.822299 11962 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:56.825628 11962 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:56.825752 11962 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:36:56.829619 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:34921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=41195
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:56.830662 11962 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:56.831755 11962 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:56.834496 11962 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:56.838356 11971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:56.838390 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:56.838346 11973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:56.838821 11962 server_base.cc:1061] running on GCE node
I20260812 06:36:56.839249 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:56.839861 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:56.841076 11962 hybrid_clock.cc:648] HybridClock initialized: now 1786516616841060 us; error 36 us; skew 500 ppm
I20260812 06:36:56.842788 11962 webserver.cc:533] Webserver started at http://127.7.212.194:41195/ using document root <none> and password file <none>
I20260812 06:36:56.843333 11962 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:56.843405 11962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:56.846958 11962 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:36:56.849136 11988 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 06:36:56.849998  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:56.849998  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.850780 11962 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:56.850873 11962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:56.851337 11962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:56.862064  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.868139 11962 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:56.868855 11962 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:56.869050 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:56.869645 11962 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:56.870839 12003 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:56.874195 11962 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:56.874259 11962 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:56.874338 11962 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:56.875914 11962 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:56.875957 11962 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:36:56.876289 12003 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap starting.
I20260812 06:36:56.884503 12003 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:56.887485 11962 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:34921
I20260812 06:36:56.887513 12222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:34921 every 8 connection(s)
I20260812 06:36:56.888353 11962 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
I20260812 06:36:56.891855  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 11962
I20260812 06:36:56.897017 12225 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:56.897260 12225 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:56.897864 12225 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:56.899864  8074 ts_manager.cc:194] Re-registered known tserver with Master: 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921)
I20260812 06:36:56.900605  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:53927
I20260812 06:36:56.904728 12003 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=7 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20260812 06:36:56.905244 12003 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap complete.
W20260812 06:36:56.905414 12003 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 12ms of latency in SegmentAllocator::Sync()
I20260812 06:36:56.917812 12003 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent bootstrapping tablet: real 0.042s	user 0.025s	sys 0.004s
I20260812 06:36:56.920059 12003 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.922446 12003 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Initialized, Role: FOLLOWER
I20260812 06:36:56.922951 12003 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 8.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.923591 12225 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:56.923863 12003 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260812 06:36:56.931098 11668 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:56.931263 11668 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.932732 11668 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.938041 11918 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:56.938169 11918 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.938408 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.938701 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 8.
I20260812 06:36:56.939620  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.939818 11918 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.940413 11840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:56.940568 11840 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 for term 9 because replica has last-logged OpId of term: 8 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 34.
I20260812 06:36:56.941890 11931 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:56.942013 11931 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.942842 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.942998 11306 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 for term 9 because replica has last-logged OpId of term: 8 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 34.
I20260812 06:36:56.943480 11931 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.945963 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.946106 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 8.
I20260812 06:36:56.947948 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.948072 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 8.
I20260812 06:36:56.948683  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.949030 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.949165 11572 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 8.
I20260812 06:36:56.949213 11464 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:56.949472 11668 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 06:36:56.949479 11720 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60; no voters: 
I20260812 06:36:56.949826 11918 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 06:36:56.949896 11918 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:56.949939 11918 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:56.950454  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.951206 11918 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.951851 11918 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 9 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.952484 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.953066 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:56.953189 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:56.954201 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 9.
I20260812 06:36:56.955101 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:56.955251 12133 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:56.955765  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.956279 11925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:56.956321 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:56.956372 11925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.956431 11572 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:56.956795  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:56.957050 11925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:56.957626 11572 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 9.
I20260812 06:36:56.958007 11840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:56.958206 11840 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.959336  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:56.959434  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:56.959530  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:56.960325 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.960427  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 9.
I20260812 06:36:56.960464 11572 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.961723  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:56.961903 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.962330 10798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.962515 10797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:56.962821 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:56.962944 11720 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 9e8291da04bf4d8dafd875d0dfe291f0
I20260812 06:36:56.963138 11918 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Leader election won for term 9
I20260812 06:36:56.963299 11918 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 LEADER]: Becoming Leader. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Running, Role: LEADER
I20260812 06:36:56.963650 11918 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 8.35, Last appended by leader: 35, Current term: 9, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:56.963936  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:56.964021 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 9.
I20260812 06:36:56.964372 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:56.964632 11931 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260812 06:36:56.965361 11936 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 21ms of latency in SegmentAllocator::Sync()
I20260812 06:36:56.966816 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.967013 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.967085  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a reported cstate change: term changed from 8 to 9, leader changed from 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194) to 1adfc3d9d367476090a9513b769ac76a (127.7.212.196). New cstate: current_term: 9 leader_uuid: "1adfc3d9d367476090a9513b769ac76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:56.967978 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:36:56.970167 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:56.971155 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:56.971312 11572 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.972446 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:56.972599 11306 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.972972 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:56.973003  9391 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:56.973129 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:56.973227 11925 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 06:36:56.973780 11840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:56.978233 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.018090 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.030828 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.033972 11929 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9f32be8bb322423eaa290fd71a8baf10)
I20260812 06:36:57.034125 11929 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:57.035207 11512 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46932: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.035794 11929 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.041348 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:57.041538 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:57.042359 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:57.042412 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.042518 11572 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:57.042614 11306 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
W20260812 06:36:57.043115 11918 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:57.043174 11918 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:57.043900  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:57.045055 11840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:57.045351  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20260812 06:36:57.045611 10678 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a60c871d36ed4e6ea4b74231f1414739; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:57.045922 11929 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 8 FOLLOWER]: Advancing to term 9
I20260812 06:36:57.045881 11918 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 34, Time since last communication: 0.000s
I20260812 06:36:57.046268  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:57.046448  9511 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 9: Already voted for candidate 1adfc3d9d367476090a9513b769ac76a in this term.
I20260812 06:36:57.046561  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:57.047034 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 13ms of latency in SegmentAllocator::Sync()
I20260812 06:36:57.047088 11929 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260812 06:36:57.058183 11572 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 8 index: 34. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20260812 06:36:57.058719 11918 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:57.060170 12133 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 8 index: 33. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20260812 06:36:57.060829 12293 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:57.063454 12303 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 32ms of latency in SegmentAllocator::Sync()
W20260812 06:36:57.071729  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.074918  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20260812 06:36:57.075361 12301 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:57.076367 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 79ms of latency in SegmentAllocator::Sync()
E20260812 06:36:57.082471 12301 fault_injection.cc:59] injecting fault for kudu (pid 11691): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:57.101781 11995 connection.cc:573] server connection from 127.7.212.196:40327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:57.102139  9818 connection.cc:573] client connection to 127.7.212.196:38991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:57.102272  9818 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:57.110146  8019 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 06:36:57.110541  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.196:38991
--local_ip_for_outbound_sockets=127.7.212.196
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=40525
--webserver_interface=127.7.212.196
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:57.127038 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.155876  9391 connection.cc:466] server connection from 127.7.212.196:43951 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.196:43951 (request call id 3) could send its response
W20260812 06:36:57.161321 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.164235 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.186177  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.228253 12304 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:57.228456 12304 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:57.228473 12304 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:57.228488 12304 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:57.228528 12304 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:57.228552 12304 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:57.228595 12304 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:57.232012 12304 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:57.232122 12304 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.196
I20260812 06:36:57.235916 12304 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.196:38991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.7.212.196
--webserver_port=40525
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.12304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.196

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:57.237087 12304 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:57.238145 12304 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:57.241130 12304 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:57.244783 12313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.244796 12311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.244797 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:57.245204 12304 server_base.cc:1061] running on GCE node
I20260812 06:36:57.246021 12304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:57.246752 12304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:57.247936 12304 hybrid_clock.cc:648] HybridClock initialized: now 1786516617247914 us; error 37 us; skew 500 ppm
I20260812 06:36:57.249994 12304 webserver.cc:533] Webserver started at http://127.7.212.196:40525/ using document root <none> and password file <none>
I20260812 06:36:57.250619 12304 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:57.250684 12304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:57.254292 12304 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.002s
I20260812 06:36:57.256424 12325 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:57.257597 12304 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:57.257758 12304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:57.258149 12304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:57.270328 12304 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:57.270952 12304 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:57.271157 12304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:57.271749 12304 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:57.273063 12339 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:57.276605 12304 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:57.276674 12304 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:36:57.276795 12304 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:57.278451 12304 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:57.278527 12304 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:36:57.278820 12339 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap starting.
W20260812 06:36:57.286085  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.288075 12339 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Log is configured to *not* fsync() on all Append() calls
W20260812 06:36:57.289661  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.291265 12304 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.196:38991
I20260812 06:36:57.291344 12525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.196:38991 every 8 connection(s)
I20260812 06:36:57.292243 12304 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 06:36:57.297333  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 12304
I20260812 06:36:57.301453 12526 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:57.301690 12526 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:57.302275 12526 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
W20260812 06:36:57.304100  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.304219  8074 ts_manager.cc:194] Re-registered known tserver with Master: 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991)
I20260812 06:36:57.305182  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.196:57887
I20260812 06:36:57.309167 12339 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 12 replicates
I20260812 06:36:57.309574 12339 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap complete.
W20260812 06:36:57.309700 12339 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 93ms of latency in SegmentAllocator::Sync()
I20260812 06:36:57.347352 11961 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:57.347532 11961 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.348510 11961 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:57.349022 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.349074 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:57.349222 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 9.
I20260812 06:36:57.349503 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.349691 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8291da04bf4d8dafd875d0dfe291f0 in term 9.
I20260812 06:36:57.350346  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:36:57.351779  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.353879 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
W20260812 06:36:57.355086  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.356416 12404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:57.358174  8214 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:57.358286  8214 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:57.358604 11961 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 06:36:57.360842 11668 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:57.360981 11668 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.361855 11668 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.362440  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:57.362465  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.362576  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 9.
I20260812 06:36:57.362871 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.363029 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 9.
I20260812 06:36:57.363085  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:57.363242 12231 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:57.363354 12231 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.363517 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.363655 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 9.
I20260812 06:36:57.364003 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:57.364146 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 9.
I20260812 06:36:57.364253 11464 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 
I20260812 06:36:57.364524 11668 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 06:36:57.364588 11668 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:57.364637 11668 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.365224 12231 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.365962 11668 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.366808 11668 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.366989 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:57.367115 10796 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.367481  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:57.368248 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:57.368350 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 10.
I20260812 06:36:57.368371 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.368891 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:57.369021 12133 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.369011 12429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:57.369024 12404 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:57.369577 11464 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 election: Tablet error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:57.369767 11464 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:57.370183 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 10.
I20260812 06:36:57.370731  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:57.370865  8331 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.371855  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 10.
I20260812 06:36:57.372833 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 10.
I20260812 06:36:57.373167 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.373260 11464 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: 1adfc3d9d367476090a9513b769ac76a, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:57.373329 10796 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.373518 11668 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Leader election won for term 10
I20260812 06:36:57.373735 11668 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:36:57.374095 11668 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.377393 11925 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:36:57.377548 11925 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.377786  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 9 to 10, leader changed from 1adfc3d9d367476090a9513b769ac76a (127.7.212.196) to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 10 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:57.378381 11925 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.378455 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:57.378602 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.378722 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.378866 11306 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.379400 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.379405  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:57.379544 10796 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.379554  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.379801 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:57.380262  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:57.380295  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:57.380393  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 9.
I20260812 06:36:57.380427  8331 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.380898 11572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:57.381096  9387 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:57.381279 11925 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.382351 11925 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 06:36:57.384855 12429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:57.385269  9387 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:57.386425 12429 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:57.386883 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
W20260812 06:36:57.386883 11989 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:57.386998 11989 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739
I20260812 06:36:57.387059 11306 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 10: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:36:57.387231 12231 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 06:36:57.403141 12339 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent bootstrapping tablet: real 0.125s	user 0.027s	sys 0.004s
I20260812 06:36:57.405000 12339 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Replica starting. Triggering 12 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.407087 12339 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Initialized, Role: FOLLOWER
I20260812 06:36:57.407630 12339 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.408331 12526 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:57.408607 12339 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
W20260812 06:36:57.417971 12035 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.421262 11668 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:57.421271 11933 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 66ms of latency in SegmentAllocator::Sync()
W20260812 06:36:57.421329 11668 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 06:36:57.421767 11668 fault_injection.cc:59] injecting fault for kudu (pid 11437): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 06:36:57.428376 11306 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
I20260812 06:36:57.434155  8331 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
I20260812 06:36:57.434947 12429 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 9 FOLLOWER]: Advancing to term 10
I20260812 06:36:57.438194 12429 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
W20260812 06:36:57.445888  9818 connection.cc:573] client connection to 127.7.212.195:40613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:57.445996 12328 connection.cc:573] server connection from 127.7.212.195:59987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:57.446193  8217 connection.cc:573] server connection from 127.7.212.195:41857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:57.446188 11174 connection.cc:573] server connection from 127.7.212.195:47957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:57.454013  8019 ts_itest-base.cc:575] TS 2 appears to have crashed. Restarting.
I20260812 06:36:57.454528  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:57.485387 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.509543 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.511924 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.559525 12371 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52230: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:57.568369 12557 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:57.568607 12557 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:57.568653 12557 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:57.568714 12557 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:57.568819 12557 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:57.568876 12557 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:57.568969 12557 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:57.572309 12557 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:57.572499 12557 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
W20260812 06:36:57.574265 12371 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52230: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:57.576905 12557 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.12557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:57.577999 12557 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:57.579276 12557 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:57.579677 12371 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52230: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
E20260812 06:36:57.582327 12557 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:57.586079 12567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.586126 12568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.586196 12570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:57.586385 12557 server_base.cc:1061] running on GCE node
I20260812 06:36:57.586917 12557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:57.587462 12557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:57.588682 12557 hybrid_clock.cc:648] HybridClock initialized: now 1786516617588669 us; error 33 us; skew 500 ppm
I20260812 06:36:57.590377 12557 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:57.591051 12557 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:57.591112 12557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:57.594440 12557 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:36:57.596400 12581 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:57.597528 12557 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:57.597672 12557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:57.598060 12557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:57.624373 12557 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:57.625206 12557 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:57.625442 12557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:57.626219 12557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:57.627806 12593 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 06:36:57.628545 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.631803 12557 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:57.631899 12557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20260812 06:36:57.631985 12557 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:57.633667 12557 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:57.633740 12557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:57.634335 12593 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
W20260812 06:36:57.636566 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.644150 12593 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
W20260812 06:36:57.645998 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.646263 12557 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:36:57.646306 12784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:57.647346 12557 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:57.651592  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 12557
I20260812 06:36:57.657284 12786 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:57.657567 12786 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:57.658279 12786 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:57.660588  8074 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:57.661753  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:56055
I20260812 06:36:57.668843 12593 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 14 replicates
I20260812 06:36:57.669430 12593 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
W20260812 06:36:57.669636 12593 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 11ms of latency in SegmentAllocator::Sync()
I20260812 06:36:57.681155 12593 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.047s	user 0.031s	sys 0.005s
I20260812 06:36:57.683516 12593 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Replica starting. Triggering 14 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.685883 12593 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:57.686371 12593 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 10.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.687091 12786 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:57.687332 12593 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
W20260812 06:36:57.702121 12600 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41028: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.702690 12549 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:57.702920 12549 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.704500 12549 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
W20260812 06:36:57.704849 12600 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41028: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:57.709748 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.709939 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 10.
I20260812 06:36:57.710103 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.710278 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 10.
I20260812 06:36:57.711246 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:57.711422 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 10.
I20260812 06:36:57.711822 12330 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 
I20260812 06:36:57.712087 12549 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 06:36:57.712146 12549 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:57.712195 12549 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.713591 12549 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.714072 12699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:57.714213 12549 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 11 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.714257  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:57.714390 12699 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a for term 11 because replica has last-logged OpId of term: 10 index: 38, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 36.
I20260812 06:36:57.714409  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 10.
I20260812 06:36:57.714577  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:57.714684  9510 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.714809 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:57.714855 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:57.714924 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.714979 12133 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.715764  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 11.
I20260812 06:36:57.715903 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 11.
I20260812 06:36:57.716013 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 11.
I20260812 06:36:57.716454 12699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:57.716595 12699 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.716599 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:57.716722 10796 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.716737 12328 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:57.716989 12549 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 FOLLOWER]: Leader election won for term 11
I20260812 06:36:57.717181 12549 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 LEADER]: Becoming Leader. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Running, Role: LEADER
I20260812 06:36:57.717334  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:57.717466  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 10.
I20260812 06:36:57.717510 12549 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 9.36, Last appended by leader: 36, Current term: 11, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.717671  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 11 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:57.717788  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 10 FOLLOWER]: Advancing to term 11
I20260812 06:36:57.717864 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 11.
I20260812 06:36:57.718681  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 11.
I20260812 06:36:57.718907 12699 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1adfc3d9d367476090a9513b769ac76a for term 11 because replica has last-logged OpId of term: 10 index: 38, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 36.
W20260812 06:36:57.719974 12813 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 45ms of latency in SegmentAllocator::Sync()
I20260812 06:36:57.721081  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a reported cstate change: term changed from 10 to 11, leader changed from 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195) to 1adfc3d9d367476090a9513b769ac76a (127.7.212.196). New cstate: current_term: 11 leader_uuid: "1adfc3d9d367476090a9513b769ac76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:57.771390 12035 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.771390 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.794480 12549 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:57.794562 12549 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:57.795674  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 11 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
I20260812 06:36:57.796271 12549 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 24, Time since last communication: 0.000s
E20260812 06:36:57.796663 12549 fault_injection.cc:59] injecting fault for kudu (pid 12304): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 06:36:57.815217  8019 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 06:36:57.815635  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.196:38991
--local_ip_for_outbound_sockets=127.7.212.196
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=40525
--webserver_interface=127.7.212.196
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:57.843801 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.903614 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.914538  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.931375 12815 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:57.931602 12815 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:57.931636 12815 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:57.931689 12815 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:57.931757 12815 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:57.931818 12815 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:57.931892 12815 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:57.935290 12815 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:57.935456 12815 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.196
I20260812 06:36:57.939503 12815 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.196:38991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.7.212.196
--webserver_port=40525
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.12815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.196

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:57.940649 12815 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:57.941854 12815 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:57.944607 12815 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 06:36:57.948622 12798 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260812 06:36:57.948750 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.948650 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.948648 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:57.948930 12798 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.949266 12815 server_base.cc:1061] running on GCE node
I20260812 06:36:57.949632 12815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:57.950201 12815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:57.950728 12798 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.951401 12815 hybrid_clock.cc:648] HybridClock initialized: now 1786516617951388 us; error 30 us; skew 500 ppm
W20260812 06:36:57.953403 12585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:36:57.953660 12815 webserver.cc:533] Webserver started at http://127.7.212.196:40525/ using document root <none> and password file <none>
I20260812 06:36:57.954363 12815 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:57.954443 12815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:57.957319  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:57.957513  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 11.
I20260812 06:36:57.958068 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:57.958240 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 11.
I20260812 06:36:57.958966 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:57.959108 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 11.
I20260812 06:36:57.959468 12587 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:57.959766 12798 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260812 06:36:57.959838 12798 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:57.959880 12798 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.960136 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:57.960258 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 11.
I20260812 06:36:57.961154 12798 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:57.961300 12585 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:36:57.961612  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:57.961763  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 11.
I20260812 06:36:57.961884 12798 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:57.962383 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:57.962435 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:57.962482 12815 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.002s	sys 0.003s
I20260812 06:36:57.962517 12133 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.962550 11306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.962932  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:57.963052  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.963600 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 12.
I20260812 06:36:57.963625 11306 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 12.
I20260812 06:36:57.963881  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:57.964010  9510 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.964172  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 12.
I20260812 06:36:57.964711 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 12 candidate_status { last_received { term: 10 index: 38 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:57.964886 10796 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:57.965154  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 12.
W20260812 06:36:57.965466 12585 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:36:57.965627 12585 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a
I20260812 06:36:57.965685 12845 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:57.965778 12798 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Leader election won for term 12
I20260812 06:36:57.965953 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 12.
I20260812 06:36:57.965988 12798 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:36:57.966342 12798 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 10.38, Last appended by leader: 38, Current term: 12, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:57.967000 12815 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:57.967136 12815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:57.967662 12815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:57.970072  8074 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 11 to 12, leader changed from 1adfc3d9d367476090a9513b769ac76a (127.7.212.196) to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 12 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:57.972138  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:57.986092 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.011186 12815 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:58.011965 12815 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:58.012185 12815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:58.012898 12815 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:58.014209 12862 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:58.017721 12815 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:58.017818 12815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20260812 06:36:58.017904 12815 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:58.019536 12815 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:58.019606 12815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:36:58.020048 12862 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap starting.
I20260812 06:36:58.029250 12862 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:58.031389 12815 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.196:38991
I20260812 06:36:58.031411 13082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.196:38991 every 8 connection(s)
I20260812 06:36:58.032408 12815 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 06:36:58.035555  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 12815
I20260812 06:36:58.041754 13083 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:58.042027 13083 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:58.042690 13083 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
W20260812 06:36:58.044374 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.044751  8074 ts_manager.cc:194] Re-registered known tserver with Master: 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991)
I20260812 06:36:58.045637  8074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.196:50377
I20260812 06:36:58.051050 12862 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=24 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 13 replicates
I20260812 06:36:58.051513 12862 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap complete.
I20260812 06:36:58.051867 12862 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent bootstrapping tablet: real 0.032s	user 0.029s	sys 0.000s
I20260812 06:36:58.053915 12862 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 FOLLOWER]: Replica starting. Triggering 13 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:58.055238 12798 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:58.055308 12798 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:58.056121  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.056344 12862 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Initialized, Role: FOLLOWER
I20260812 06:36:58.056828 12798 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:58.056970 12862 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 11.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.057646 13083 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:58.057950 12862 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
W20260812 06:36:58.059361 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 33ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.059877  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.065327 12981 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 11 FOLLOWER]: Advancing to term 12
I20260812 06:36:58.068094 12981 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 11 index: 37. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.069533 12798 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:58.069732 12798 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Peer 1adfc3d9d367476090a9513b769ac76a log is divergent from this leader: its last log entry 11.37 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260812 06:36:58.070163  9510 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.070753 12846 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:58.072125 11306 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.072628 12798 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:58.073257 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 78ms of latency in SegmentAllocator::Sync()
I20260812 06:36:58.073768 12981 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Deduplicated request from leader. Original: 3.24->[3.25-12.39]   Dedup: 9.36->[10.37-12.39]
I20260812 06:36:58.074000 12981 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting all ops after (but not including) 36
I20260812 06:36:58.074091 12981 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting uncommitted NO_OP operation due to leader change: 11.37
I20260812 06:36:58.074187 12981 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 06:36:58.076529 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 8ms of latency in SegmentAllocator::Sync()
I20260812 06:36:58.078591 12133 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.079032 10796 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 8 index: 34. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20260812 06:36:58.079087 12798 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
I20260812 06:36:58.079495 12846 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 24, Time since last communication: 0.000s
W20260812 06:36:58.082501 13111 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 06:36:58.088420 13088 mvcc.cc:204] Tried to move back new op lower bound from 7317572061742116864 to 7317572055633514496. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054351110144}]
I20260812 06:36:58.089236 12793 mvcc.cc:204] Tried to move back new op lower bound from 7317572058940731392 to 7317572055633514496. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054333382656}]
I20260812 06:36:58.094578 13109 mvcc.cc:204] Tried to move back new op lower bound from 7317572064954982400 to 7317572055633514496. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054336425984}]
I20260812 06:36:58.095022 13103 mvcc.cc:204] Tried to move back new op lower bound from 7317572064954982400 to 7317572055633514496. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054333382656}]
W20260812 06:36:58.097078  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.097656  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.097836  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.101456  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.103659  8255 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.104218 12035 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.104236 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.104238 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.105558 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.106496 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.106957 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.107846 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.108556 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.110307 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.110551 13110 mvcc.cc:204] Tried to move back new op lower bound from 7317572064954982400 to 7317572055633514496. Current Snapshot: MvccSnapshot[applied={T|T < 7317572054336425984}]
W20260812 06:36:58.110886 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.111539 13171 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51712 (ReqId={client: 4105b259dcfe4c81b5e5a4faec14f6a3, seq_no=24, attempt_no=37}) took 1369 ms. Trace:
I20260812 06:36:58.111598 13171 rpcz_store.cc:276] 0812 06:36:56.742359 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:36:56.742430 (+    71us) service_pool.cc:224] Handling call
0812 06:36:58.111515 (+1369085us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260812 06:36:58.115103 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 141ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.115329 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 47ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.115675 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 208ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.116403 12303 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.116979 13111 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 47ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.117033 12847 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 6ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.123517 12847 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 71ms of latency in SegmentAllocator::Sync()
I20260812 06:36:58.155606 13098 mvcc.cc:204] Tried to move back new op lower bound from 7317572061742116864 to 7317572058907107328. Current Snapshot: MvccSnapshot[applied={T|T < 7317572058940731392}]
I20260812 06:36:58.156813 13190 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39090 (ReqId={client: 4105b259dcfe4c81b5e5a4faec14f6a3, seq_no=30, attempt_no=49}) took 1398 ms. Trace:
I20260812 06:36:58.156896 13190 rpcz_store.cc:276] 0812 06:36:56.758066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:36:56.758145 (+    79us) service_pool.cc:224] Handling call
0812 06:36:58.156787 (+1398642us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260812 06:36:58.160882 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 65ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.165773 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 06:36:58.170275  9438 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39090 (request call id 64) took 1013 ms. Trace:
I20260812 06:36:58.170389  9438 rpcz_store.cc:276] 0812 06:36:57.157287 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:36:57.157357 (+    70us) service_pool.cc:224] Handling call
0812 06:36:57.157533 (+   176us) tablet_service.cc:2890] Created scanner ed53c706bd6d47c2a334a45b5f37812b for tablet 2cd3bf4d446147de95e618239dd3074c, query id is 22deccacb2c645f994ae5f0fb699ef3b
0812 06:36:57.157784 (+   251us) tablet_service.cc:3030] Creating iterator
0812 06:36:57.157811 (+    27us) tablet_service.cc:3408] Waiting safe time to advance
0812 06:36:58.072907 (+915096us) tablet_service.cc:3415] Waiting for operations to commit
0812 06:36:58.160372 (+ 87465us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1002545 microseconds
0812 06:36:58.160569 (+   197us) tablet_service.cc:3055] Iterator created
0812 06:36:58.162881 (+  2312us) tablet_service.cc:3077] Iterator init: OK
0812 06:36:58.162953 (+    72us) tablet_service.cc:3120] has_more: true
0812 06:36:58.163041 (+    88us) tablet_service.cc:3137] Continuing scan request
0812 06:36:58.163172 (+   131us) tablet_service.cc:3201] Found scanner ed53c706bd6d47c2a334a45b5f37812b for tablet 2cd3bf4d446147de95e618239dd3074c, query id is 22deccacb2c645f994ae5f0fb699ef3b
0812 06:36:58.170262 (+  7090us) inbound_call.cc:177] Queueing success response
Metrics: {"compiler_manager_pool.queue_time_us":238,"rowset_iterators":1,"scanner_bytes_read":0,"thread_start_us":157,"threads_started":1}
W20260812 06:36:58.171128  9944 scanner-internal.cc:458] Time spent opening tablet: real 3.740s	user 0.003s	sys 0.001s
W20260812 06:36:58.189446 12303 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 2ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.191444 13111 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 115ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.193547 12303 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 136ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.195614 12847 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 9ms of latency in SegmentAllocator::Sync()
E20260812 06:36:58.202800 13134 fault_injection.cc:59] injecting fault for kudu (pid 12557): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:58.218427  9818 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613) has failed: Network error: recv got EOF from 127.7.212.195:40613 (error 108)
W20260812 06:36:58.220999 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.221105  8019 ts_itest-base.cc:575] TS 2 appears to have crashed. Restarting.
I20260812 06:36:58.221800  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:58.222237 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.224974 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.225502 10729 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.226143 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.226236 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.226797  9391 connection.cc:466] server connection from 127.7.212.195:56243 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:56243 (request call id 5) could send its response
W20260812 06:36:58.227182 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.228379 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.229564 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.232038 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.233292 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.234269 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.234511  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.235590  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.235635  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.238463  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.239025  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.241694  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.246912 12901 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.247440 12897 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.247877 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.246912 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.249269 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.250133 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.255511 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.255511 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.256662  8215 connection.cc:466] server connection from 127.7.212.195:36083 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:36083 (request call id 5) could send its response
W20260812 06:36:58.256685 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.257750 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.259014 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.259601 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.265000  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.266049  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.267132  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.267155  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.270426  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.282014  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.307060 10683 connection.cc:466] server connection from 127.7.212.195:43931 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:43931 (request call id 9) could send its response
W20260812 06:36:58.309564 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.310649 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.310652 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.312816 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.318077 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.321799 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.321799 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.324995 12899 tablet_service.cc:3080] error setting up scanner 2e0b8b82a2064e849c187c072f87239d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.210s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
W20260812 06:36:58.325910 11174 connection.cc:466] server connection from 127.7.212.195:60961 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:60961 (request call id 5) could send its response
W20260812 06:36:58.326452 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.327662 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.327962 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.330127 11991 connection.cc:466] server connection from 127.7.212.195:57351 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.195:57351 (request call id 7) could send its response
W20260812 06:36:58.333396 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.334510 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.334764  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.338754 12899 tablet_service.cc:3080] error setting up scanner 63dd3aede2ed4c4884610140f52f04b3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.224s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
W20260812 06:36:58.343423  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.344676  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.344879  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.348317  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.350968  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.352329 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.359826 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.361070 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.361351 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.363590 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.364672 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.371095 12892 tablet_service.cc:3080] error setting up scanner c571784074274d84a884c3846f909be7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.256s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
W20260812 06:36:58.371287 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.377887 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.379539 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.381007 13194 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:58.381201 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.381263 13194 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:58.381289 13194 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:58.381309 13194 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:58.381361 13194 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:58.381386 13194 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:58.381445 13194 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:58.382961 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.384258 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.386174 13194 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:58.386319 13194 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
W20260812 06:36:58.387986  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.390710 13194 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:58.391995 13194 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:58.393441 13194 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:58.396421  8254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
E20260812 06:36:58.396721 13194 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:58.396839  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.398697  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.400934  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.401207 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.401450 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.402400  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.402597 13194 server_base.cc:1061] running on GCE node
W20260812 06:36:58.403079 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:58.403576 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:58.404181 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:58.405367 13194 hybrid_clock.cc:648] HybridClock initialized: now 1786516618405346 us; error 42 us; skew 500 ppm
I20260812 06:36:58.407454 13194 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:36:58.408138 13194 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:58.408260 13194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:58.412117 13194 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:58.414260 13215 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:58.415334 13194 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:58.415422 13194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:58.415913 13194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:58.430517 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.432008 13194 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:58.432672 13194 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:58.432914 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:58.433580 13194 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:58.434974 13228 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 06:36:58.435335 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.438630 13194 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:58.438699 13194 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20260812 06:36:58.438763 13194 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 06:36:58.439970 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.440696 13194 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:58.440752 13194 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
W20260812 06:36:58.440863 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.441680 13228 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
W20260812 06:36:58.448975 10728 tablet_service.cc:3080] error setting up scanner 521224f014f647e687e8166bed3b95a6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.261s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
I20260812 06:36:58.453075 13194 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:36:58.454391 13194 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:36:58.454838 13409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:36:58.455561 13228 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
W20260812 06:36:58.457935 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.459986 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.460126 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.460690 13191 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.460861 13191 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.461621 13191 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.462313 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.462689 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.463160 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.463299  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 13194
I20260812 06:36:58.463574  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:58.463655 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.464035  9387 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
W20260812 06:36:58.464253 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.464252 13191 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260812 06:36:58.465121 13132 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.465289 13132 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.466882 13132 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.467819 13412 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:58.468070 13412 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:58.468693 13412 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:58.470312 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.470494 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 12.
W20260812 06:36:58.470779 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.471918  8077 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:36:58.472893  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:55417
I20260812 06:36:58.473057 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.477939  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.478112  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 12.
W20260812 06:36:58.478195  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.480013 13328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.482394 13328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:36:58.482882  9438 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.482966 12849 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:58.483246  9387 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:58.482892  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.484030  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
W20260812 06:36:58.484891  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.486756 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.487185 12849 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:58.487488 13132 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260812 06:36:58.487609 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 13 candidate_status { last_received { term: 12 index: 49 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
W20260812 06:36:58.488150  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.494459  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.494889 13228 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=42 ignored=0} inserts{seen=32 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20260812 06:36:58.495795 13228 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
W20260812 06:36:58.495990 13228 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 116ms of latency in SegmentAllocator::Sync()
W20260812 06:36:58.501149 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.504518 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.505684 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.509974 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.511053 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:58.515957 13160 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.516078 13160 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.516907 13160 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:36:58.517174 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.517524 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.517623 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.518074  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.518253  9510 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 for term 13 because replica has last-logged OpId of term: 12 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
W20260812 06:36:58.518291 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:58.518625  8214 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0; no voters: 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:58.518800 13160 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260812 06:36:58.521919 13328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.522055 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.522256 12981 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 for term 13 because replica has last-logged OpId of term: 12 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
W20260812 06:36:58.522336  8214 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:58.526862 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.527837 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.530058 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.534286 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.535442 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.540658 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.551390  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.553251  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.554307  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.558611  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.560752  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.563905  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.580821 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.587359 10728 tablet_service.cc:3080] error setting up scanner 8e291e4d6a7449a8aa8e72df0de7153d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.400s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
I20260812 06:36:58.592501 13176 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.592656 13176 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.593303 13176 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.593709 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.593888 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 12.
I20260812 06:36:58.593966 13140 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.594074 13140 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.594903 13140 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.595049 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.595142  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.595285  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 12.
I20260812 06:36:58.595922 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.595922  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:58.596051 10796 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 12.
I20260812 06:36:58.596072  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 12.
I20260812 06:36:58.596207 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.596331  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:58.596444  8331 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 12.
I20260812 06:36:58.596964  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.597091  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 12.
I20260812 06:36:58.597390 10678 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:58.597452 11989 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:58.597604 13176 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260812 06:36:58.597656 13176 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.597631 13140 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260812 06:36:58.597694 13176 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.597692 13140 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.597728 13140 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.598835 13176 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.598838 13140 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.599406 13140 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 election: Requested vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.599409 13176 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.600302 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:58.600440 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a60c871d36ed4e6ea4b74231f1414739 in current term 13: Already voted for candidate 9f32be8bb322423eaa290fd71a8baf10 in this term.
I20260812 06:36:58.600682 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:58.600844 10796 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.600919 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:58.601241  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:58.601361  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.601331 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:58.602411  9511 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:58.602669  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:58.602794  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.602881  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:58.602790  8331 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:58.603006  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.603103 13328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.603430 13333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:58.603495 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.603435 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:58.603886  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 13.
W20260812 06:36:58.604036 11989 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:58.604027 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
W20260812 06:36:58.604143 11989 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:58.604161 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 12.
I20260812 06:36:58.604295 11989 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:58.604421 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:58.604476 13140 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260812 06:36:58.604537 12980 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 12 FOLLOWER]: Advancing to term 13
W20260812 06:36:58.604702 10678 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:58.604962 10678 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 election: Tablet error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:58.605014 10678 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60
I20260812 06:36:58.605170 13176 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260812 06:36:58.605778 12980 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:58.606642 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.606642 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:58.606832 12980 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
W20260812 06:36:58.611243 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.611284 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.612323 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.612434 13228 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.171s	user 0.045s	sys 0.000s
I20260812 06:36:58.614387 13228 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.616041 13461 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 89c37a5fdd994ff58d6a23cdafbc6209)
I20260812 06:36:58.616199 13461 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.617010 13461 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.617089 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.617228 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 13: Already voted for candidate 9f32be8bb322423eaa290fd71a8baf10 in this term.
I20260812 06:36:58.617660  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.617825  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.618237 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.618307  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:58.618412 10796 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.618505  8330 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 13: Already voted for candidate 9f32be8bb322423eaa290fd71a8baf10 in this term.
I20260812 06:36:58.618847 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:58.618930 13228 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:36:58.619335 13461 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.619457 13228 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 12.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.620163 13412 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:58.620443 13228 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260812 06:36:58.620510 13461 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260812 06:36:58.620754 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.624377 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.624591 13327 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 13 because replica has last-logged OpId of term: 12 index: 51, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.625168 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 13 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.625337 12981 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
W20260812 06:36:58.636386 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.638732 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.638692 10728 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.641049 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.642331 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.647513 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.665060 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.666157  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.668444  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.671693  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.672447  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.675496  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.693286  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.696904 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.700150 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.700287 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.703330 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.706354 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.722677 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.727303 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.729399 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.730417 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.734619 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.737839 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.754462 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.758054  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.760120  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.761397  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.766551  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.768831  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.781385 10727 tablet_service.cc:3080] error setting up scanner 4907ad3ca9e74fd48b024f4e3cdb4a07 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.594s (maximum allowed 0.150s): new_scan_request { tablet_id: "2cd3bf4d446147de95e618239dd3074c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317572068024623104 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\261\253\360\323\006\"3\n\005slave\022*\n 383db6ea515e4292abfa510513502daa\020\001\030\001 \001(\001" signature: "<redacted>""*\201d\242!m/Y\303\3430\035T\247mo\213-\350y\337\310-\252\216Sz\325\276\036F\t^Qc8x#\256L\273+\000\346\314:]\361\300A\247\225\350\270\">X\371\242\017\352r\003\230" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ab1e4fc0e258494fafa5b951ab0f5bf5"
W20260812 06:36:58.789656  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.790870 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.794672 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.795734 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.800041 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.802258 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.821944 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.824313 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.826658 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.832024 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.833770 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.839793 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.858410 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.859357 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.864029 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.867055 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.870294 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.874419 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.893160 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.895282 12033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.900969  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.902891  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.906200  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.909390  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.928089  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.933188  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
I20260812 06:36:58.935703 13461 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:58.935885 13461 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.936561 13461 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.936894 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.937019 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.937130 12133 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 14 because replica has last-logged OpId of term: 12 index: 51, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.937213 10796 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 14 because replica has last-logged OpId of term: 12 index: 51, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.937356 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.937453 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:58.937548 12981 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 14 because replica has last-logged OpId of term: 12 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.937526  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:58.937618 13327 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 14 because replica has last-logged OpId of term: 12 index: 51, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.937657  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 13.
I20260812 06:36:58.937743  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 14 candidate_status { last_received { term: 12 index: 40 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.937911  9510 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c74db17d01e8469fadd613ce468bbc60 for term 14 because replica has last-logged OpId of term: 12 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 40.
I20260812 06:36:58.938146 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739
I20260812 06:36:58.938407 13461 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260812 06:36:58.939107 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.939107 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.940356 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.945607 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:58.956352 13468 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:58.956553 13468 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:58.958148 13468 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:58.963759 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:58.963809 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:58.963977 12133 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 13: Already voted for candidate 9f32be8bb322423eaa290fd71a8baf10 in this term.
I20260812 06:36:58.963958 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 13.
I20260812 06:36:58.965766 10796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:58.965958 10796 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.966018  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:58.966174  9510 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
W20260812 06:36:58.966933 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:58.967636 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:58.967806 12981 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 13: Already voted for candidate a60c871d36ed4e6ea4b74231f1414739 in this term.
I20260812 06:36:58.968107 13216 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:58.968287  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 13 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:58.968417 13468 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 12 FOLLOWER]: Advancing to term 13
I20260812 06:36:58.968461  8330 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 13: Already voted for candidate 9f32be8bb322423eaa290fd71a8baf10 in this term.
I20260812 06:36:58.970383 13468 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260812 06:36:58.971351 12892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:58.975791 10727 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51712: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.977103 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.977103 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:58.986367 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.003981 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.008073 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.014587 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.014626  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.015667  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:59.025367 13176 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:59.025662 13176 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.026361 13176 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.026886 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:59.026872  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:59.027031  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:59.026988 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:59.027045 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:59.027153 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:59.027227 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:59.027370 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:59.027524 10683 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578; no voters: 
I20260812 06:36:59.027665 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:59.027808 13176 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260812 06:36:59.027824 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
I20260812 06:36:59.027863 13176 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:59.027900 13176 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.027925  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.028056  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 13.
W20260812 06:36:59.028452  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:59.029067 13176 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.029740 13176 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 14 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.029933 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:59.029951 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:36:59.030010 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:59.030054 13327 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.030076 12133 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.030119 12981 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.030220  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:59.030331  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.031131 12133 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.031144 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.031144 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.031354  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.031488  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:59.031636  9510 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.031863 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 14 candidate_status { last_received { term: 12 index: 51 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:59.031890 10678 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: 
I20260812 06:36:59.031980 11308 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 13 FOLLOWER]: Advancing to term 14
I20260812 06:36:59.032420 13176 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 FOLLOWER]: Leader election won for term 14
I20260812 06:36:59.032613 13176 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 LEADER]: Becoming Leader. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Running, Role: LEADER
I20260812 06:36:59.032797  9510 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.033051 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 14.
I20260812 06:36:59.033023 13176 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 12.51, Last appended by leader: 51, Current term: 14, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:36:59.034564 13111 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 75ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.035873  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 reported cstate change: term changed from 12 to 14, leader changed from 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195) to a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197). New cstate: current_term: 14 leader_uuid: "a60c871d36ed4e6ea4b74231f1414739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:59.042347  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.048542  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.052899  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.055619 13176 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:59.055682 13176 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:59.056605  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 40. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.056728 12981 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 49. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.056900 12133 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 51. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.057379 11308 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 40. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.057425  9510 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 49. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.057580 13327 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 12 index: 51. Preceding OpId from leader: term: 14 index: 53. (index mismatch)
I20260812 06:36:59.058010 13176 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 39, Time since last communication: 0.000s
I20260812 06:36:59.058193 13480 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 39, Time since last communication: 0.000s
I20260812 06:36:59.058363 13494 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 42, Time since last communication: 0.000s
I20260812 06:36:59.058633 13480 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 39, Time since last communication: 0.000s
I20260812 06:36:59.058750 13489 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 39, Time since last communication: 0.000s
I20260812 06:36:59.058890 13490 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 42, Time since last communication: 0.000s
W20260812 06:36:59.061954 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 70ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.063382 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 134ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.065881 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 90ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.066100 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 181ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.069952 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 74ms of latency in SegmentAllocator::Sync()
E20260812 06:36:59.091457 13480 fault_injection.cc:59] injecting fault for kudu (pid 10651): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:59.114368  9818 connection.cc:573] client connection to 127.7.212.197:41571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:36:59.114567  9944 meta_cache.cc:1510] marking tablet server a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571) as failed
I20260812 06:36:59.120076  8019 ts_itest-base.cc:575] TS 4 appears to have crashed. Restarting.
I20260812 06:36:59.120498  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.197:41571
--local_ip_for_outbound_sockets=127.7.212.197
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=33771
--webserver_interface=127.7.212.197
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:36:59.132725 13218 connection.cc:466] server connection from 127.7.212.197:41159 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:41159 (request call id 5) could send its response
W20260812 06:36:59.144590 12855 connection.cc:466] server connection from 127.7.212.197:47211 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:47211 (request call id 6) could send its response
W20260812 06:36:59.156136 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.156591 11173 connection.cc:466] server connection from 127.7.212.197:49855 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:49855 (request call id 6) could send its response
W20260812 06:36:59.198043  9387 connection.cc:466] server connection from 127.7.212.197:54499 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:54499 (request call id 6) could send its response
W20260812 06:36:59.199105 13254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.199097 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.200399 13262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.200400 13254 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41114: Illegal state: replica 89c37a5fdd994ff58d6a23cdafbc6209 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.232734 13529 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:59.232985 13529 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:59.233016 13529 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:59.233053 13529 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:59.233122 13529 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:59.233172 13529 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:59.233253 13529 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:59.236490 13529 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:59.236653 13529 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.197
I20260812 06:36:59.240682 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.197:41571
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.7.212.197
--webserver_port=33771
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.197

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
W20260812 06:36:59.241298 12034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
I20260812 06:36:59.242033 13529 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:59.243202 13529 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:59.244665  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
E20260812 06:36:59.246225 13529 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:59.247586  8217 connection.cc:466] server connection from 127.7.212.197:52885 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:52885 (request call id 7) could send its response
W20260812 06:36:59.250119 13542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:59.250119 13540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:59.250351 13529 server_base.cc:1061] running on GCE node
W20260812 06:36:59.250181 13538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:59.251116 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:59.251724 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:59.252926 13529 hybrid_clock.cc:648] HybridClock initialized: now 1786516619252914 us; error 30 us; skew 500 ppm
I20260812 06:36:59.254820 13529 webserver.cc:533] Webserver started at http://127.7.212.197:33771/ using document root <none> and password file <none>
I20260812 06:36:59.255532 13529 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:59.255597 13529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:59.259421 13529 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:59.261497 13550 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:59.262538 13529 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:59.262668 13529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:36:59.263149 13529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:59.279356 13529 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:59.280117 13529 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:59.280349 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:59.281060 13529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:59.282436 13561 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:59.285770 13529 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:59.285843 13529 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20260812 06:36:59.285933 13529 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:59.287622 13529 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:59.287742 13529 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:36:59.288123 13561 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap starting.
W20260812 06:36:59.288383  9437 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39090: Illegal state: replica d12989a6d0f445359c9b09b7ce6cb578 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.291111 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
I20260812 06:36:59.299068 13561 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:59.300845 13529 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.197:41571
I20260812 06:36:59.300912 13748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.197:41571 every 8 connection(s)
I20260812 06:36:59.301796 13529 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 06:36:59.308060  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 13529
I20260812 06:36:59.311827 13749 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:59.312073 13749 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:59.312566 13749 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:59.314446  8077 ts_manager.cc:194] Re-registered known tserver with Master: a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571)
I20260812 06:36:59.315445  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.197:50665
I20260812 06:36:59.326541 13561 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=7 applied=41 ignored=0} inserts{seen=31 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 06:36:59.327029 13561 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap complete.
W20260812 06:36:59.327260 13561 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 67ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.335171 12899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52252: Illegal state: replica 1adfc3d9d367476090a9513b769ac76a is not leader of this config: current role FOLLOWER
W20260812 06:36:59.335170 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
I20260812 06:36:59.360414 13468 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:59.360569 13468 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.360857 13461 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:59.361020 13461 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.361665 13468 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.361786 13461 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.362387 12133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:59.362423  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.362816 12130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:59.363132  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.363343 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:59.363382 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:59.363673  9510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:59.363842 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:59.363965 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 14.
I20260812 06:36:59.364199  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:59.364517 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:59.364639 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 14.
I20260812 06:36:59.365177 13219 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:59.365381 13468 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260812 06:36:59.365492 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:59.365667 13461 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260812 06:36:59.370067 13516 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:59.370182 13516 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.369040 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:59.369060 13671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 15 candidate_status { last_received { term: 14 index: 53 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:59.370954 13516 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.371609 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:59.371760 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 14.
W20260812 06:36:59.371821 13216 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 15 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:59.374868 12130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:36:59.374982  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.374994 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:59.375115 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 14.
I20260812 06:36:59.375388  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:59.379429 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 15 candidate_status { last_received { term: 14 index: 55 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:36:59.379853 12849 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 15 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:59.379961 12849 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, c74db17d01e8469fadd613ce468bbc60; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:59.380151 13516 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260812 06:36:59.380918 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.383540  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
I20260812 06:36:59.386511 13140 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:59.386626 13140 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.387430 13140 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.387842 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:59.388001 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 14.
I20260812 06:36:59.388121  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:36:59.388134 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:59.388263 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:59.388293 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 14.
I20260812 06:36:59.388397 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 14.
I20260812 06:36:59.388739 11989 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60; no voters: d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:59.388731  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.388944 13140 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260812 06:36:59.389012 13140 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:36:59.389068 13140 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.390419 13140 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.391259 13140 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 election: Requested vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.391573  9511 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:36:59.391707 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:59.391812 11308 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.391856 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:59.391974 12981 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.392818 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 15.
I20260812 06:36:59.393157 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 15.
I20260812 06:36:59.393517 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:59.393566 13671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:59.393846 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:59.394006 13327 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 14 FOLLOWER]: Advancing to term 15
W20260812 06:36:59.393992 11989 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:36:59.394114 11989 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:36:59.394074  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 15 candidate_status { last_received { term: 14 index: 56 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:59.394670 13561 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent bootstrapping tablet: real 0.107s	user 0.039s	sys 0.000s
I20260812 06:36:59.395176 13327 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f32be8bb322423eaa290fd71a8baf10 in term 15.
I20260812 06:36:59.395725 11989 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:36:59.396064 13140 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 FOLLOWER]: Leader election won for term 15
I20260812 06:36:59.396315 13140 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 LEADER]: Becoming Leader. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Running, Role: LEADER
I20260812 06:36:59.396629 13140 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 14.56, Last appended by leader: 56, Current term: 15, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.396726 13561 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.399694 13561 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Initialized, Role: FOLLOWER
I20260812 06:36:59.399837  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 reported cstate change: term changed from 14 to 15, leader changed from a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197) to 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194). New cstate: current_term: 15 leader_uuid: "9f32be8bb322423eaa290fd71a8baf10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:36:59.400233 13561 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 14.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.401072 13749 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:36:59.401330 13561 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260812 06:36:59.427989  8256 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54898: Illegal state: replica 9e8291da04bf4d8dafd875d0dfe291f0 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.431474 12303 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 109ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.431535 13140 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:59.431595 13140 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:59.432503 11308 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 53. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.432503 12981 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 55. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.432741  9511 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.432909  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.433027 13672 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 14 FOLLOWER]: Advancing to term 15
I20260812 06:36:59.433518 13327 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 53. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.434037  9511 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 54. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.434163  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 54. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.434165 13773 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.434614 13773 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.434753 13140 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.435452 13672 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 15 FOLLOWER]: Refusing update from remote peer 9f32be8bb322423eaa290fd71a8baf10: Log matching property violated. Preceding OpId in replica: term: 14 index: 52. Preceding OpId from leader: term: 15 index: 58. (index mismatch)
I20260812 06:36:59.434969 13784 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.436323 13785 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.436681 13781 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 41, Time since last communication: 0.000s
E20260812 06:36:59.436867 13773 fault_injection.cc:59] injecting fault for kudu (pid 11962): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:36:59.437000 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 16ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.447575 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 52ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.454250 13218 connection.cc:573] server connection from 127.7.212.194:60365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:36:59.454873  9823 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921) has failed: Network error: recv got EOF from 127.7.212.194:34921 (error 108)
I20260812 06:36:59.462458  8019 ts_itest-base.cc:575] TS 1 appears to have crashed. Restarting.
I20260812 06:36:59.462661 13498 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:59.462792 13498 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.462925  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:34921
--local_ip_for_outbound_sockets=127.7.212.194
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=41195
--webserver_interface=127.7.212.194
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:36:59.463727 13498 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.465402 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:36:59.465747 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:36:59.465921 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 15.
I20260812 06:36:59.466490 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:36:59.467224  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:36:59.467409  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 15.
W20260812 06:36:59.468693  9391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.194:34921: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260812 06:36:59.469017 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:36:59.469277 13672 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 15.
I20260812 06:36:59.469615  9387 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209
I20260812 06:36:59.469852 13498 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260812 06:36:59.469983 13498 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:59.470044 13498 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 15 FOLLOWER]: Advancing to term 16
W20260812 06:36:59.470216  9391 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Network error: Client connection negotiation failed: client connection to 127.7.212.194:34921: connect: Connection refused (error 111)
I20260812 06:36:59.471504 13498 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.472445 13498 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:36:59.472857 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:36:59.472883 13327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:36:59.473160 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:36:59.473315 13672 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 15 FOLLOWER]: Advancing to term 16
I20260812 06:36:59.473553  8330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:36:59.473723  8330 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 15 FOLLOWER]: Advancing to term 16
I20260812 06:36:59.474421 13672 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 16.
I20260812 06:36:59.475059  8330 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 16.
I20260812 06:36:59.475075 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 16 candidate_status { last_received { term: 14 index: 54 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:36:59.475226 11308 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 15 FOLLOWER]: Advancing to term 16
I20260812 06:36:59.476292 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 16.
I20260812 06:36:59.476697  9391 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209
I20260812 06:36:59.476882 13498 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Leader election won for term 16
I20260812 06:36:59.477119 13498 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 LEADER]: Becoming Leader. State: Replica: d12989a6d0f445359c9b09b7ce6cb578, State: Running, Role: LEADER
W20260812 06:36:59.477257 13602 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.477479  9391 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Network error: Client connection negotiation failed: client connection to 127.7.212.194:34921: connect: Connection refused (error 111)
I20260812 06:36:59.477495 13498 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 14.54, Last appended by leader: 54, Current term: 16, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.480993  8072 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 reported cstate change: term changed from 15 to 16, leader changed from 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194) to d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198). New cstate: current_term: 16 leader_uuid: "d12989a6d0f445359c9b09b7ce6cb578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:36:59.494462 13602 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.497473 13602 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.500204 12850 connection.cc:466] server connection from 127.7.212.194:50637 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.194:50637 (request call id 5) could send its response
W20260812 06:36:59.501822 13602 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.501828 13603 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.503994 13603 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:36:59.533387 13498 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:36:59.533458 13498 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:36:59.534318  8330 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Refusing update from remote peer d12989a6d0f445359c9b09b7ce6cb578: Log matching property violated. Preceding OpId in replica: term: 14 index: 54. Preceding OpId from leader: term: 16 index: 56. (index mismatch)
I20260812 06:36:59.534719 11308 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Refusing update from remote peer d12989a6d0f445359c9b09b7ce6cb578: Log matching property violated. Preceding OpId in replica: term: 14 index: 53. Preceding OpId from leader: term: 16 index: 56. (index mismatch)
I20260812 06:36:59.535256 12981 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 15 FOLLOWER]: Advancing to term 16
I20260812 06:36:59.535373 13327 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 15 FOLLOWER]: Advancing to term 16
W20260812 06:36:59.535615  9391 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 -> Peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Couldn't send request to peer 9f32be8bb322423eaa290fd71a8baf10. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.194:34921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:36:59.535769 13797 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.536556 12981 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 16 FOLLOWER]: Refusing update from remote peer d12989a6d0f445359c9b09b7ce6cb578: Log matching property violated. Preceding OpId in replica: term: 15 index: 58. Preceding OpId from leader: term: 16 index: 56. (term mismatch)
I20260812 06:36:59.536612 12981 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting all ops after (but not including) 55
I20260812 06:36:59.536636 12981 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting uncommitted WRITE_OP operation due to leader change: 14.56
I20260812 06:36:59.536626 13327 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Refusing update from remote peer d12989a6d0f445359c9b09b7ce6cb578: Log matching property violated. Preceding OpId in replica: term: 15 index: 58. Preceding OpId from leader: term: 16 index: 56. (term mismatch)
I20260812 06:36:59.536689 13327 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting all ops after (but not including) 55
I20260812 06:36:59.536749 13327 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting uncommitted WRITE_OP operation due to leader change: 14.56
I20260812 06:36:59.536909 12981 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting uncommitted NO_OP operation due to leader change: 15.57
I20260812 06:36:59.536958 12981 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 06:36:59.537019 12981 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting uncommitted WRITE_OP operation due to leader change: 15.58
I20260812 06:36:59.537096 13327 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting uncommitted NO_OP operation due to leader change: 15.57
I20260812 06:36:59.537145 13327 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 06:36:59.537189 13327 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting uncommitted WRITE_OP operation due to leader change: 15.58
I20260812 06:36:59.537863 13672 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Refusing update from remote peer d12989a6d0f445359c9b09b7ce6cb578: Log matching property violated. Preceding OpId in replica: term: 14 index: 52. Preceding OpId from leader: term: 16 index: 56. (index mismatch)
I20260812 06:36:59.539131 13498 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 42, Time since last communication: 0.000s
I20260812 06:36:59.539258 13498 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Peer 89c37a5fdd994ff58d6a23cdafbc6209 log is divergent from this leader: its last log entry 14.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20260812 06:36:59.539299 13797 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.540333 13327 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 12.42->[12.43-16.56]   Dedup: 14.54->[16.55-16.56]
I20260812 06:36:59.540436 13327 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting all ops after (but not including) 54
I20260812 06:36:59.540454 13327 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Aborting uncommitted WRITE_OP operation due to leader change: 14.55
I20260812 06:36:59.539693 13793 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 41, Time since last communication: 0.000s
I20260812 06:36:59.541481 13498 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 42, Time since last communication: 0.000s
I20260812 06:36:59.541591 13498 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [LEADER]: Peer 1adfc3d9d367476090a9513b769ac76a log is divergent from this leader: its last log entry 14.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
W20260812 06:36:59.544895 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 57ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.548751 12981 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 16 FOLLOWER]: Deduplicated request from leader. Original: 12.42->[12.43-16.56]   Dedup: 14.54->[16.55-16.56]
I20260812 06:36:59.548879 12981 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting all ops after (but not including) 54
I20260812 06:36:59.548907 12981 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Aborting uncommitted WRITE_OP operation due to leader change: 14.55
W20260812 06:36:59.551064 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.551159 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 9ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.553790 13462 mvcc.cc:204] Tried to move back new op lower bound from 7317572073605844992 to 7317572071564034048. Current Snapshot: MvccSnapshot[applied={T|T < 7317572067812282368}]
I20260812 06:36:59.562670 13520 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52252 (ReqId={client: 4105b259dcfe4c81b5e5a4faec14f6a3, seq_no=26, attempt_no=73}) took 1192 ms. Trace:
I20260812 06:36:59.562748 13520 rpcz_store.cc:276] 0812 06:36:58.370433 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:36:58.370533 (+   100us) service_pool.cc:224] Handling call
0812 06:36:59.562636 (+1192103us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260812 06:36:59.566244  9944 scanner-internal.cc:458] Time spent opening tablet: real 1.244s	user 0.004s	sys 0.000s
I20260812 06:36:59.568203  8255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:54898 (request call id 56) took 1450 ms. Trace:
I20260812 06:36:59.568284  8255 rpcz_store.cc:276] 0812 06:36:58.117629 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:36:58.117695 (+    66us) service_pool.cc:224] Handling call
0812 06:36:58.117813 (+   118us) tablet_service.cc:2890] Created scanner 97a2dae4026e41b5a42b7941f85e0b7f for tablet 2cd3bf4d446147de95e618239dd3074c, query id is c434212af27d4c76a02c1ce9e78bf0e9
0812 06:36:58.117971 (+   158us) tablet_service.cc:3030] Creating iterator
0812 06:36:58.117989 (+    18us) tablet_service.cc:3408] Waiting safe time to advance
0812 06:36:59.062983 (+944994us) tablet_service.cc:3415] Waiting for operations to commit
0812 06:36:59.564826 (+501843us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1446825 microseconds
0812 06:36:59.564968 (+   142us) tablet_service.cc:3055] Iterator created
0812 06:36:59.566607 (+  1639us) tablet_service.cc:3077] Iterator init: OK
0812 06:36:59.566670 (+    63us) tablet_service.cc:3120] has_more: true
0812 06:36:59.566755 (+    85us) tablet_service.cc:3137] Continuing scan request
0812 06:36:59.566846 (+    91us) tablet_service.cc:3201] Found scanner 97a2dae4026e41b5a42b7941f85e0b7f for tablet 2cd3bf4d446147de95e618239dd3074c, query id is c434212af27d4c76a02c1ce9e78bf0e9
0812 06:36:59.568187 (+  1341us) inbound_call.cc:177] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"thread_start_us":106,"threads_started":1}
W20260812 06:36:59.569684 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 127ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.570647  9947 scanner-internal.cc:458] Time spent opening tablet: real 5.136s	user 0.004s	sys 0.000s
W20260812 06:36:59.571986 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 47ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.572999 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 5ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.574368 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 13ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.580242 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 80ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.600382 13787 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:36:59.600585 13787 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:36:59.600666 13787 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:36:59.600734 13787 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:36:59.600850 13787 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:59.600906 13787 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:59.600992 13787 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:59.606292 13787 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:59.606523 13787 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:36:59.608177 13775 mvcc.cc:204] Tried to move back new op lower bound from 7317572073605844992 to 7317572073385058304. Current Snapshot: MvccSnapshot[applied={T|T < 7317572067792265216}]
I20260812 06:36:59.611609 13787 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:34921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=41195
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.13787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:59.613111 13787 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:59.614543 13787 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:36:59.618170 13787 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:36:59.620805 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 72ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.627702 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 57ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.629866 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:59.632947 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:59.633992 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 50ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.634289 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 94ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.635418 13787 server_base.cc:1061] running on GCE node
W20260812 06:36:59.635509 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:59.636288 13787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:59.637013 13787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:59.640334 13787 hybrid_clock.cc:648] HybridClock initialized: now 1786516619640269 us; error 75 us; skew 500 ppm
I20260812 06:36:59.642426 13787 webserver.cc:533] Webserver started at http://127.7.212.194:41195/ using document root <none> and password file <none>
I20260812 06:36:59.643024 13787 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:59.643112 13787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:59.650030 13787 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.002s	sys 0.001s
I20260812 06:36:59.652293 13867 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:59.653680 13787 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:59.653832 13787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:36:59.654289 13787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:36:59.666887 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 56ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.677856 13787 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:59.678661 13787 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:59.678906 13787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:59.679600 13787 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:59.681209 13880 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:36:59.685470 13787 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:36:59.685563 13787 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
I20260812 06:36:59.685647 13787 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:36:59.688037 13787 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:36:59.688548 13787 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
W20260812 06:36:59.695452 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 85ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.696292 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.688484 13880 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap starting.
W20260812 06:36:59.707090 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 74ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.707389 13880 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:59.713675 13787 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:34921
I20260812 06:36:59.714795 13787 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
I20260812 06:36:59.715519 14066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:34921 every 8 connection(s)
I20260812 06:36:59.716629  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 13787
W20260812 06:36:59.733510 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 42ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.739872 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 62ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.743105 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 55ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.747067 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 51ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.758908 13880 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=7 applied=42 ignored=0} inserts{seen=32 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 16 replicates
I20260812 06:36:59.759689 13880 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap complete.
I20260812 06:36:59.761118 13880 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent bootstrapping tablet: real 0.073s	user 0.048s	sys 0.000s
I20260812 06:36:59.764709 13880 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 FOLLOWER]: Replica starting. Triggering 16 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.768307 14068 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:36:59.768549 14068 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:59.769140 14068 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:36:59.771494 13880 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Initialized, Role: FOLLOWER
I20260812 06:36:59.772140 13880 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 15.58, Last appended by leader: 58, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:36:59.774560  8077 ts_manager.cc:194] Re-registered known tserver with Master: 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921)
I20260812 06:36:59.775241 13880 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.005s
I20260812 06:36:59.775588  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:34821
I20260812 06:36:59.777698 14068 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
W20260812 06:36:59.785589 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 36ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.799257 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.800860 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 26ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.801182 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 27ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.806890 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 33ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.814201 13990 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 15 FOLLOWER]: Advancing to term 16
I20260812 06:36:59.816704 13990 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting all ops after (but not including) 54
I20260812 06:36:59.819228 13990 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 14.55
W20260812 06:36:59.826630 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 26ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.832859 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 108ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.838796 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 2ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.843874 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.853706 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 2ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.867347 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 16ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.868462 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 34ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.873840 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 85ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.875727 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 103ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.880267 13990 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 14.56
I20260812 06:36:59.880554 13990 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted NO_OP operation due to leader change: 15.57
I20260812 06:36:59.880627 13990 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 06:36:59.880694 13990 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Aborting uncommitted WRITE_OP operation due to leader change: 15.58
W20260812 06:36:59.886734 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 45ms of latency in SegmentAllocator::Sync()
I20260812 06:36:59.889001 14072 mvcc.cc:204] Tried to move back new op lower bound from 7317572074278039552 to 7317572073385058304. Current Snapshot: MvccSnapshot[applied={T|T < 7317572067808493568}]
W20260812 06:36:59.890023 14092 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 32ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.916450 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 42ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.930584 14092 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 41ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.945186 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 103ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.963826 11430 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 191ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.973523 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 135ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.980496 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 107ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.981319 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 27ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.983511 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 06:36:59.986902 14092 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 164ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.011104 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 39ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.061086 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 41ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.065567 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.089115 11928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 73ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.095925 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 32ms of latency in SegmentAllocator::Sync()
E20260812 06:37:00.115085 13498 fault_injection.cc:59] injecting fault for kudu (pid 9365): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 06:37:00.132328 11173 connection.cc:573] server connection from 127.7.212.198:36157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:37:00.133033 13552 connection.cc:573] server connection from 127.7.212.198:54977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:37:00.135277  8019 ts_itest-base.cc:575] TS 5 appears to have crashed. Restarting.
I20260812 06:37:00.135747  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.198:35139
--local_ip_for_outbound_sockets=127.7.212.198
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=35371
--webserver_interface=127.7.212.198
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:37:00.143781 13922 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.143900 13921 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.144127 13917 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.143781 13898 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.144930 13902 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.145056 13892 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.145226 13884 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.145816 13911 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.145929 13899 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57208: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.150110 13605 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.151051 13605 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.151726 13875 connection.cc:466] server connection from 127.7.212.198:52233 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.198:52233 (request call id 3) could send its response
W20260812 06:37:00.153522 13605 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.154471 13601 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39814: Illegal state: replica a60c871d36ed4e6ea4b74231f1414739 is not leader of this config: current role FOLLOWER
W20260812 06:37:00.155802  8217 connection.cc:466] server connection from 127.7.212.198:56999 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.198:56999 (request call id 23) could send its response
W20260812 06:37:00.250871 14095 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 06:37:00.251096 14095 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:00.251127 14095 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:00.251160 14095 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:00.251236 14095 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:00.251288 14095 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:00.251365 14095 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:00.255455 14095 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:00.255589 14095 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.198
I20260812 06:37:00.259665 14095 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.198:35139
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.7.212.198
--webserver_port=35371
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.14095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.198

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:00.260684 14095 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:00.261869 14095 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:00.264683 14095 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:00.269229 14105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.269260 14114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.269616 14107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:00.270128 14095 server_base.cc:1061] running on GCE node
I20260812 06:37:00.270641 14095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:00.271528 14095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:00.272812 14095 hybrid_clock.cc:648] HybridClock initialized: now 1786516620272754 us; error 72 us; skew 500 ppm
I20260812 06:37:00.275463 14095 webserver.cc:533] Webserver started at http://127.7.212.198:35371/ using document root <none> and password file <none>
I20260812 06:37:00.276116 14095 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:00.276199 14095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:00.279982 14095 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:00.282325 14122 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:00.283427 14095 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:00.283573 14095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "d12989a6d0f445359c9b09b7ce6cb578"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:37:00.284056 14095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:00.298067 14095 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:00.298776 14095 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:00.298993 14095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:00.299613 14095 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:00.301158 14132 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:00.305234 14095 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:00.305310 14095 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:37:00.305375 14095 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:00.307592 14095 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:00.307658 14095 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:00.308106 14132 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap starting.
I20260812 06:37:00.319746 14095 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.198:35139
I20260812 06:37:00.320953 14324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.198:35139 every 8 connection(s)
I20260812 06:37:00.321204 14095 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 06:37:00.323858 14132 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:00.323905  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 14095
I20260812 06:37:00.332139 14325 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:00.332430 14325 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:00.333119 14325 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:00.334200  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 8188
I20260812 06:37:00.344571  8077 ts_manager.cc:194] Re-registered known tserver with Master: d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139)
I20260812 06:37:00.345652  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.198:38543
I20260812 06:37:00.364125 14334 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d12989a6d0f445359c9b09b7ce6cb578)
I20260812 06:37:00.364302 14334 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.368432 14334 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.374101 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 17 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:37:00.377245 14351 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:00.368200 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.193:37797 (local address 127.7.212.197:50557)
0812 06:37:00.373746 (+  5546us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:00.373763 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 06:37:00.373920 (+   157us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 06:37:00.376563 (+  2643us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5485,"thread_start_us":103,"threads_started":1}
W20260812 06:37:00.377518 13552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:37:00.377743 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 17 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.378589  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:37797
--local_ip_for_outbound_sockets=127.7.212.193
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=33451
--webserver_interface=127.7.212.193
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:37:00.378868 13552 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.380095 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 17 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.380568 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 17 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:00.381103 13555 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a60c871d36ed4e6ea4b74231f1414739; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:00.381381 14334 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 06:37:00.387986 14359 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d12989a6d0f445359c9b09b7ce6cb578)
I20260812 06:37:00.388243 14359 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.387352 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 17 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:37:00.390017 13552 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 17 pre-election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.394168 14359 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.396741 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 17 candidate_status { last_received { term: 16 index: 104 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:00.396976 13672 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 for term 17 because replica has last-logged OpId of term: 16 index: 109, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 104.
I20260812 06:37:00.399680 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 17 candidate_status { last_received { term: 16 index: 104 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:00.401093 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 17 candidate_status { last_received { term: 16 index: 104 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:37:00.401667 13868 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 17 pre-election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:37:00.402372 13868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.404516 14378 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d12989a6d0f445359c9b09b7ce6cb578)
I20260812 06:37:00.404639 14378 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.405516 14378 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.406808 12980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:00.407014 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 17 candidate_status { last_received { term: 16 index: 104 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:00.407851 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:00.408031 13672 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 16.
I20260812 06:37:00.408291 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:00.408429 13325 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 16.
I20260812 06:37:00.409227 13868 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:37:00.409821 14359 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260812 06:37:00.411708 11172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.412422 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
W20260812 06:37:00.412662 11172 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
W20260812 06:37:00.413057 11172 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 pre-election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.415023 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.415218 13990 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 16.
I20260812 06:37:00.415586 11174 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: 1adfc3d9d367476090a9513b769ac76a, 9e8291da04bf4d8dafd875d0dfe291f0, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:37:00.415864 14378 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260812 06:37:00.415934 14378 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Starting leader election (detected failure of leader d12989a6d0f445359c9b09b7ce6cb578)
I20260812 06:37:00.416005 14378 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 16 FOLLOWER]: Advancing to term 17
W20260812 06:37:00.416327 13868 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.416560 14386 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader d12989a6d0f445359c9b09b7ce6cb578)
I20260812 06:37:00.416740 14386 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.417336 14378 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.418128 14378 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.419540 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:37:00.419713 13672 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 16 FOLLOWER]: Advancing to term 17
I20260812 06:37:00.420660 13672 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 17.
I20260812 06:37:00.421214 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:37:00.421351 12981 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 16 FOLLOWER]: Advancing to term 17
I20260812 06:37:00.422302 12981 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 17.
I20260812 06:37:00.422618 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:37:00.422973 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:37:00.423079 13325 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 16 FOLLOWER]: Advancing to term 17
I20260812 06:37:00.423874 13325 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 17.
W20260812 06:37:00.424443 11172 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 election: Tablet error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.424602 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60; no voters: d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:37:00.425076 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 17 candidate_status { last_received { term: 16 index: 104 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.425076 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:37:00.425207 13990 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 16 FOLLOWER]: Advancing to term 17
I20260812 06:37:00.425240 11308 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 17: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:37:00.426100 14378 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 FOLLOWER]: Leader election won for term 17
I20260812 06:37:00.426368 13990 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 17.
I20260812 06:37:00.426391 14378 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 LEADER]: Becoming Leader. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Running, Role: LEADER
I20260812 06:37:00.426783 14378 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 16.113, Last appended by leader: 113, Current term: 17, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:00.428468 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 88ms of latency in SegmentAllocator::Sync()
W20260812 06:37:00.430019 11172 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.430128  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 reported cstate change: term changed from 16 to 17, leader changed from d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198) to c74db17d01e8469fadd613ce468bbc60 (127.7.212.199). New cstate: current_term: 17 leader_uuid: "c74db17d01e8469fadd613ce468bbc60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:00.430717 14386 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.430920 14396 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:00.431032 14396 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.431905 14396 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.433197 12981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:00.433365 12981 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 17: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:37:00.434347 13671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:00.434499 13671 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 17.
I20260812 06:37:00.435076 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:00.435197 13325 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 17.
I20260812 06:37:00.436074 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.436305 14132 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap replayed 1/1 log segments. Stats: ops{read=116 overwritten=7 applied=99 ignored=0} inserts{seen=87 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
I20260812 06:37:00.436877 14132 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap complete.
I20260812 06:37:00.437139 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.437152 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.437335 13990 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 17: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:37:00.437520 14132 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent bootstrapping tablet: real 0.130s	user 0.086s	sys 0.010s
I20260812 06:37:00.437889 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:00.438094 13672 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89c37a5fdd994ff58d6a23cdafbc6209 in current term 17: Already voted for candidate c74db17d01e8469fadd613ce468bbc60 in this term.
I20260812 06:37:00.438499 13216 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209; no voters: 1adfc3d9d367476090a9513b769ac76a, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:00.438712 14386 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 06:37:00.440613 14132 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.443791 14132 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12989a6d0f445359c9b09b7ce6cb578, State: Initialized, Role: FOLLOWER
W20260812 06:37:00.444617 12849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.444542 14132 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 16.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:00.445474 13216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.445861 14132 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260812 06:37:00.446314 14325 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:00.450035 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 17 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:37:00.450439 14233 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 16.
I20260812 06:37:00.452978 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.453133 13990 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 17.
W20260812 06:37:00.453137 13216 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.456405 12855 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:00.456615 14396 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260812 06:37:00.456664 14396 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:00.456702 14396 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 17 FOLLOWER]: Advancing to term 18
I20260812 06:37:00.457908 14396 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.458629 14396 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.459699 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:37:00.459839 13672 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 17 FOLLOWER]: Advancing to term 18
I20260812 06:37:00.460956 13672 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 18.
I20260812 06:37:00.461094 14233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:37:00.461196 14232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:37:00.461241 14233 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 16.
I20260812 06:37:00.461251 13325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
I20260812 06:37:00.461367 13325 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 17 FOLLOWER]: Advancing to term 18
I20260812 06:37:00.462388 13325 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 18.
W20260812 06:37:00.463300 12849 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.463454 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:37:00.463948 13990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "1adfc3d9d367476090a9513b769ac76a" candidate_term: 18 candidate_status { last_received { term: 16 index: 113 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:37:00.464103 13990 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 17 FOLLOWER]: Advancing to term 18
W20260812 06:37:00.464510 12849 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111)
I20260812 06:37:00.465389 13990 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1adfc3d9d367476090a9513b769ac76a in term 18.
I20260812 06:37:00.465721 12855 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578
I20260812 06:37:00.466005 14396 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Leader election won for term 18
I20260812 06:37:00.466289 14396 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 LEADER]: Becoming Leader. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Running, Role: LEADER
I20260812 06:37:00.466638 14396 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 16.113, Last appended by leader: 113, Current term: 18, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:00.468570 13106 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 92ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.469993  8072 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a reported cstate change: term changed from 17 to 18, leader changed from c74db17d01e8469fadd613ce468bbc60 (127.7.212.199) to 1adfc3d9d367476090a9513b769ac76a (127.7.212.196). New cstate: current_term: 18 leader_uuid: "1adfc3d9d367476090a9513b769ac76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
W20260812 06:37:00.504810 14378 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:37:00.504930 14378 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260812 06:37:00.506590 11172 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 -> Peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Couldn't send request to peer 9e8291da04bf4d8dafd875d0dfe291f0. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:00.510742 13325 raft_consensus.cc:1240] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Rejecting Update request from peer c74db17d01e8469fadd613ce468bbc60 for earlier term 17. Current term is 18. Ops: []
I20260812 06:37:00.511364 14378 consensus_queue.cc:1059] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: INVALID_TERM, Last received: 0.0, Next index: 114, Last known committed idx: 103, Time since last communication: 0.000s
I20260812 06:37:00.511698 14378 raft_consensus.cc:3055] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 LEADER]: Stepping down as leader of term 17
I20260812 06:37:00.511775 14378 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Running, Role: LEADER
I20260812 06:37:00.512020 14378 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 17.114, Last appended by leader: 114, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.512346 14378 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 17 FOLLOWER]: Advancing to term 18
W20260812 06:37:00.522912 14356 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:00.523104 14356 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:00.523126 14356 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:00.523232 14356 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:00.523277 14356 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:00.523370 14356 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:00.526834 14356 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:00.526935 14356 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:00.530777 14356 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:37797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=33451
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.14356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:00.531769 14356 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:00.532970 14356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:00.535676 14356 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:00.539505 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.539549 14444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.539475 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:00.540494 14356 server_base.cc:1061] running on GCE node
I20260812 06:37:00.540897 14356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:00.541386 14356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:00.542583 14356 hybrid_clock.cc:648] HybridClock initialized: now 1786516620542567 us; error 34 us; skew 500 ppm
I20260812 06:37:00.544345 14356 webserver.cc:533] Webserver started at http://127.7.212.193:33451/ using document root <none> and password file <none>
W20260812 06:37:00.544889 14396 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:37:00.544948 14396 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:37:00.545047 14356 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:00.545158 14356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 06:37:00.546434 12849 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a -> Peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Couldn't send request to peer 9e8291da04bf4d8dafd875d0dfe291f0. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:00.549007 14356 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.004s	sys 0.000s
I20260812 06:37:00.551011 14462 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:00.552068 14356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:00.552217 14356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:37:00.552616 14356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:00.559876 13990 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 18 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 16 index: 104. Preceding OpId from leader: term: 18 index: 114. (index mismatch)
I20260812 06:37:00.560493 14396 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 94, Time since last communication: 0.000s
I20260812 06:37:00.562904 14356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:00.563897 14356 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:00.564097 14356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:00.564807 14356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 06:37:00.565008 14092 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 179ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.567077 14480 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:00.570830 14356 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:00.570907 14356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20260812 06:37:00.570991 14356 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:00.572557 14356 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:00.572624 14356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:00.572986 14480 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Bootstrap starting.
I20260812 06:37:00.573849 13672 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 16 index: 109. Preceding OpId from leader: term: 18 index: 114. (index mismatch)
I20260812 06:37:00.574425 14396 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 99, Time since last communication: 0.000s
W20260812 06:37:00.577339 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.578423 14232 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 16 FOLLOWER]: Advancing to term 18
I20260812 06:37:00.581298 13325 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 16 index: 113. Preceding OpId from leader: term: 18 index: 114. (index mismatch)
I20260812 06:37:00.581936 14396 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 103, Time since last communication: 0.000s
W20260812 06:37:00.583338 13506 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 50ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.583617 14232 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Refusing update from remote peer 1adfc3d9d367476090a9513b769ac76a: Log matching property violated. Preceding OpId in replica: term: 16 index: 109. Preceding OpId from leader: term: 18 index: 114. (index mismatch)
I20260812 06:37:00.584235 14396 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 99, Time since last communication: 0.000s
E20260812 06:37:00.584447 14396 fault_injection.cc:59] injecting fault for kudu (pid 12815): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 06:37:00.590118 14480 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:00.596755 14356 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:37797
I20260812 06:37:00.598045 14356 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
I20260812 06:37:00.599010 14687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:37797 every 8 connection(s)
I20260812 06:37:00.600282  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 14356
I20260812 06:37:00.600461  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 13787
W20260812 06:37:00.616360  9823 meta_cache.cc:302] tablet 2cd3bf4d446147de95e618239dd3074c: replica 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921) has failed: Network error: recv got EOF from 127.7.212.194:34921 (error 108)
I20260812 06:37:00.617637  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:34921
--local_ip_for_outbound_sockets=127.7.212.194
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=41195
--webserver_interface=127.7.212.194
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:37:00.620443 14689 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:00.620733 14689 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:00.621474 14689 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:00.623859  8072 ts_manager.cc:194] Re-registered known tserver with Master: 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.625041  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:42515
W20260812 06:37:00.633950 13218 connection.cc:466] server connection from 127.7.212.196:57955 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.196:57955 (request call id 5) could send its response
I20260812 06:37:00.650439 14480 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=89 ignored=0} inserts{seen=77 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260812 06:37:00.651062 14480 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Bootstrap complete.
W20260812 06:37:00.651266 14480 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 8ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.659732 14480 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Time spent bootstrapping tablet: real 0.087s	user 0.057s	sys 0.015s
I20260812 06:37:00.662290 14480 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.664568 14480 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8291da04bf4d8dafd875d0dfe291f0, State: Initialized, Role: FOLLOWER
I20260812 06:37:00.665144 14480 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 16.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.666057 14689 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:00.666087 14480 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
W20260812 06:37:00.732074 14693 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:00.732260 14693 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:00.732280 14693 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:00.732318 14693 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:00.732393 14693 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:00.732471 14693 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:00.735711 14693 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:00.735863 14693 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:00.739583 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:34921
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=41195
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:00.740583 14693 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:00.741660 14693 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:00.744360 14693 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:00.748201 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.748232 14704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.748255 14708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:00.748590 14693 server_base.cc:1061] running on GCE node
I20260812 06:37:00.748966 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:00.749481 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:00.750646 14693 hybrid_clock.cc:648] HybridClock initialized: now 1786516620750628 us; error 29 us; skew 500 ppm
I20260812 06:37:00.752352 14693 webserver.cc:533] Webserver started at http://127.7.212.194:41195/ using document root <none> and password file <none>
I20260812 06:37:00.752979 14693 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:00.753036 14693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:00.756237 14693 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:37:00.758256 14725 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:00.759274 14693 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:00.759394 14693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "9f32be8bb322423eaa290fd71a8baf10"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:37:00.759773 14693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:00.774016 14693 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:00.774583 14693 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:00.774799 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:00.775394 14693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:00.776638 14745 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:00.779964 14693 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:00.780047 14693 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:37:00.780124 14693 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:00.781737 14693 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:00.781808 14693 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:00.782137 14745 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap starting.
I20260812 06:37:00.793665 14745 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:00.794405 14693 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:34921
I20260812 06:37:00.794508 14945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:34921 every 8 connection(s)
I20260812 06:37:00.795336 14693 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
I20260812 06:37:00.795928  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 14693
I20260812 06:37:00.796114  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 13194
I20260812 06:37:00.823721 14949 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:00.823983 14949 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:00.824594 14949 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:00.825014 14423 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:00.825130 14423 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.826741 14423 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.828138  8072 ts_manager.cc:194] Re-registered known tserver with Master: 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921)
I20260812 06:37:00.828989  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:40613
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=42853
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:37:00.829202  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:37673
W20260812 06:37:00.833180 14957 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:00.828002 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.195:40613 (local address 127.7.212.198:53691)
0812 06:37:00.832382 (+  4380us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:00.832450 (+    68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":4316,"thread_start_us":97,"threads_started":1}
W20260812 06:37:00.833349 14124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection reset by peer (error 104)
I20260812 06:37:00.835137 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 19 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.835332 11308 raft_consensus.cc:2410] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d12989a6d0f445359c9b09b7ce6cb578 for term 19 because replica has last-logged OpId of term: 17 index: 114, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 109.
W20260812 06:37:00.836812 14124 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
I20260812 06:37:00.837368 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 19 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:37:00.837877 14124 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:37:00.839047 14124 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:00.837949 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 19 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.839309 14423 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260812 06:37:00.840160 14128 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 19 pre-election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.841457 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 19 candidate_status { last_received { term: 16 index: 109 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:00.841861 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 16.
I20260812 06:37:00.852887 14378 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:00.853027 14378 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.853763 14378 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.855895 13672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 19 candidate_status { last_received { term: 17 index: 114 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:00.856274 14232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 19 candidate_status { last_received { term: 17 index: 114 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:37:00.856426 14232 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 18.
I20260812 06:37:00.858615 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 19 candidate_status { last_received { term: 17 index: 114 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
W20260812 06:37:00.859366 11174 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 19 pre-election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:37:00.859555 11172 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
W20260812 06:37:00.859664 11172 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:37:00.859699 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739
I20260812 06:37:00.859863 14378 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260812 06:37:00.860249 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 19 candidate_status { last_received { term: 17 index: 114 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:00.860421 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 16.
I20260812 06:37:00.876881 14334 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:37:00.877044 14334 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.877766 14334 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.878644 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:00.878813 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 18.
I20260812 06:37:00.880640 14232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:37:00.880821 14232 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 18.
W20260812 06:37:00.881428 13552 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
W20260812 06:37:00.881537 13552 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:37:00.884789 14745 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap replayed 1/1 log segments. Stats: ops{read=125 overwritten=11 applied=102 ignored=0} inserts{seen=90 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 12 replicates
I20260812 06:37:00.885352 14745 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Bootstrap complete.
W20260812 06:37:00.885550 14745 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 53ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.885543 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:00.885696 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 16.
I20260812 06:37:00.886052 13552 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209
I20260812 06:37:00.886287 14334 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260812 06:37:00.886349 14334 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1adfc3d9d367476090a9513b769ac76a)
I20260812 06:37:00.886404 14334 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 18 FOLLOWER]: Advancing to term 19
I20260812 06:37:00.886926 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:00.887588 14334 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:00.887943 13555 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 pre-election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.888242 14334 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:00.888588 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:37:00.888676 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:37:00.888827 11308 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 18 FOLLOWER]: Advancing to term 19
I20260812 06:37:00.889111 14232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578"
I20260812 06:37:00.889241 14232 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 18 FOLLOWER]: Advancing to term 19
I20260812 06:37:00.889921 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 19.
I20260812 06:37:00.890170 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "a60c871d36ed4e6ea4b74231f1414739" candidate_term: 19 candidate_status { last_received { term: 18 index: 114 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
W20260812 06:37:00.890347 13555 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 election: Tablet error from VoteRequest() call to peer 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:00.890337 14606 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 16 FOLLOWER]: Advancing to term 19
I20260812 06:37:00.890375 14232 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 19.
W20260812 06:37:00.892002 13552 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111)
W20260812 06:37:00.892153 13552 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111)
I20260812 06:37:00.892747 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60c871d36ed4e6ea4b74231f1414739 in term 19.
I20260812 06:37:00.893095 13552 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:37:00.893343 14334 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 FOLLOWER]: Leader election won for term 19
I20260812 06:37:00.893543 14334 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 LEADER]: Becoming Leader. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Running, Role: LEADER
I20260812 06:37:00.893870 14334 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 18.114, Last appended by leader: 114, Current term: 19, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:00.895457 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 46ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.896973  8072 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 reported cstate change: term changed from 18 to 19, leader changed from 1adfc3d9d367476090a9513b769ac76a (127.7.212.196) to a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197). New cstate: current_term: 19 leader_uuid: "a60c871d36ed4e6ea4b74231f1414739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:00.939072 14745 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent bootstrapping tablet: real 0.157s	user 0.073s	sys 0.004s
I20260812 06:37:00.941073 14745 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 18 FOLLOWER]: Replica starting. Triggering 12 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.943948 14745 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f32be8bb322423eaa290fd71a8baf10, State: Initialized, Role: FOLLOWER
I20260812 06:37:00.944470 14745 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 18.114, Last appended by leader: 114, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:00.945197 14949 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:00.945415 14745 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.002s
W20260812 06:37:00.963367 14965 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:00.963627 14965 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:00.963661 14965 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:00.963721 14965 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:00.963778 14965 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:00.963845 14965 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:00.967764 14965 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:00.967943 14965 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
W20260812 06:37:00.971711 14334 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:37:00.971784 14334 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 06:37:00.972162 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:40613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=42853
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:00.972831 14232 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 16 index: 109. Preceding OpId from leader: term: 19 index: 115. (index mismatch)
I20260812 06:37:00.973470 14965 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:00.973399 14334 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 99, Time since last communication: 0.000s
I20260812 06:37:00.974828 14965 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:00.977768 14989 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 93ms of latency in SegmentAllocator::Sync()
E20260812 06:37:00.978345 14965 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:00.982235 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.982263 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:00.982295 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:00.982648 14965 server_base.cc:1061] running on GCE node
I20260812 06:37:00.983063 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:00.983711 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:00.984864 14965 hybrid_clock.cc:648] HybridClock initialized: now 1786516620984847 us; error 28 us; skew 500 ppm
I20260812 06:37:00.985674 14863 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 18 FOLLOWER]: Advancing to term 19
I20260812 06:37:00.986667 14965 webserver.cc:533] Webserver started at http://127.7.212.195:42853/ using document root <none> and password file <none>
I20260812 06:37:00.987267 14965 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:00.987354 14965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:00.987702 14863 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 19 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 18 index: 114. Preceding OpId from leader: term: 19 index: 115. (index mismatch)
I20260812 06:37:00.988391 14334 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 102, Time since last communication: 0.000s
W20260812 06:37:00.990724 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 134ms of latency in SegmentAllocator::Sync()
I20260812 06:37:00.991587 14965 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:00.993745 15012 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:00.994742 14965 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:00.994881 14965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "89c37a5fdd994ff58d6a23cdafbc6209"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:37:00.995298 14965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:37:01.001016 13552 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 -> Peer 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): Couldn't send request to peer 1adfc3d9d367476090a9513b769ac76a. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.196:38991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:01.002152 14606 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 16 index: 94. Preceding OpId from leader: term: 19 index: 115. (index mismatch)
I20260812 06:37:01.002812 14334 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 89, Time since last communication: 0.000s
I20260812 06:37:01.013787 11308 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Refusing update from remote peer a60c871d36ed4e6ea4b74231f1414739: Log matching property violated. Preceding OpId in replica: term: 17 index: 114. Preceding OpId from leader: term: 19 index: 115. (index mismatch)
I20260812 06:37:01.014336 14334 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 103, Time since last communication: 0.000s
I20260812 06:37:01.014506 14334 consensus_queue.cc:1243] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [LEADER]: Peer c74db17d01e8469fadd613ce468bbc60 log is divergent from this leader: its last log entry 17.114 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 103
W20260812 06:37:01.014567 13552 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 -> Peer 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): Couldn't send request to peer 89c37a5fdd994ff58d6a23cdafbc6209. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.195:40613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:01.015399 11308 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 16.103->[16.104-19.115]   Dedup: 16.113->[18.114-19.115]
I20260812 06:37:01.015475 11308 pending_rounds.cc:85] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Aborting all ops after (but not including) 113
I20260812 06:37:01.015494 11308 pending_rounds.cc:107] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Aborting uncommitted NO_OP operation due to leader change: 17.114
I20260812 06:37:01.015512 11308 raft_consensus.cc:2889] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 06:37:01.016108 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.016824 14965 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:01.017485 14965 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:01.017666 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:01.018301 14965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:01.019603 15031 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:01.022964 14965 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:01.023046 14965 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:37:01.023133 14965 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:01.024550 14965 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:01.024626 14965 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:01.025113 15031 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap starting.
I20260812 06:37:01.036077 14965 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:40613
I20260812 06:37:01.036113 15227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:40613 every 8 connection(s)
I20260812 06:37:01.036949 15031 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:01.037216 14965 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 06:37:01.045576  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 14965
I20260812 06:37:01.046149  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.196:38991
--local_ip_for_outbound_sockets=127.7.212.196
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=40525
--webserver_interface=127.7.212.196
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:37:01.048334 15232 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:01.048578 15232 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:01.049201 15232 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:01.051280  8072 ts_manager.cc:194] Re-registered known tserver with Master: 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613)
I20260812 06:37:01.052412  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:57971
I20260812 06:37:01.076267 13602 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39814 (request call id 12) took 1069 ms. Trace:
I20260812 06:37:01.076385 13602 rpcz_store.cc:276] 0812 06:37:00.006882 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:37:00.006976 (+    94us) service_pool.cc:224] Handling call
0812 06:37:00.007128 (+   152us) tablet_service.cc:2890] Created scanner 6287301dbba6436384ad5cbb257a6b53 for tablet 2cd3bf4d446147de95e618239dd3074c, query id is 5c753a4fcd294fe181ee2df779bc5228
0812 06:37:00.007463 (+   335us) tablet_service.cc:3030] Creating iterator
0812 06:37:00.007490 (+    27us) tablet_service.cc:3408] Waiting safe time to advance
0812 06:37:00.060980 (+ 53490us) tablet_service.cc:3415] Waiting for operations to commit
0812 06:37:01.074854 (+1013874us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1067353 microseconds
0812 06:37:01.074937 (+    83us) tablet_service.cc:3055] Iterator created
0812 06:37:01.075540 (+   603us) tablet_service.cc:3077] Iterator init: OK
0812 06:37:01.075568 (+    28us) tablet_service.cc:3120] has_more: true
0812 06:37:01.075609 (+    41us) tablet_service.cc:3137] Continuing scan request
0812 06:37:01.075676 (+    67us) tablet_service.cc:3201] Found scanner 6287301dbba6436384ad5cbb257a6b53 for tablet 2cd3bf4d446147de95e618239dd3074c, query id is 5c753a4fcd294fe181ee2df779bc5228
0812 06:37:01.076249 (+   573us) inbound_call.cc:177] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260812 06:37:01.077744 13603 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39814 (request call id 11) took 1071 ms. Trace:
I20260812 06:37:01.077829 13603 rpcz_store.cc:276] 0812 06:37:00.006708 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:37:00.006792 (+    84us) service_pool.cc:224] Handling call
0812 06:37:00.007008 (+   216us) tablet_service.cc:2890] Created scanner d0cb54ae129a42fe9996191cea4fe6a7 for tablet 2cd3bf4d446147de95e618239dd3074c, query id is d95160a4316f4585a55dd7c4e98fa67e
0812 06:37:00.007304 (+   296us) tablet_service.cc:3030] Creating iterator
0812 06:37:00.007336 (+    32us) tablet_service.cc:3408] Waiting safe time to advance
0812 06:37:00.061522 (+ 54186us) tablet_service.cc:3415] Waiting for operations to commit
0812 06:37:01.076661 (+1015139us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1069310 microseconds
0812 06:37:01.076726 (+    65us) tablet_service.cc:3055] Iterator created
0812 06:37:01.077143 (+   417us) tablet_service.cc:3077] Iterator init: OK
0812 06:37:01.077171 (+    28us) tablet_service.cc:3120] has_more: true
0812 06:37:01.077206 (+    35us) tablet_service.cc:3137] Continuing scan request
0812 06:37:01.077250 (+    44us) tablet_service.cc:3201] Found scanner d0cb54ae129a42fe9996191cea4fe6a7 for tablet 2cd3bf4d446147de95e618239dd3074c, query id is d95160a4316f4585a55dd7c4e98fa67e
0812 06:37:01.077729 (+   479us) inbound_call.cc:177] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260812 06:37:01.079957 11226 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.080547 11229 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.081032  9944 scanner-internal.cc:458] Time spent opening tablet: real 1.075s	user 0.000s	sys 0.000s
W20260812 06:37:01.081233  9947 scanner-internal.cc:458] Time spent opening tablet: real 1.075s	user 0.001s	sys 0.000s
W20260812 06:37:01.081436 11225 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.081955 11224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.082448 11216 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.090741 11216 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.090950 11225 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.091325 11224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.092324 11224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.096711 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 56ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.096877 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 50ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.100085 14989 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.103590 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 45ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.114494 15031 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap replayed 1/1 log segments. Stats: ops{read=118 overwritten=4 applied=103 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 06:37:01.114964 15031 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Bootstrap complete.
W20260812 06:37:01.115137 15031 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.133671 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 8ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.138135 14989 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 49ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.147845 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 14ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.152233 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 48ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.158371 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 145ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.160511 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 58ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.162691 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 11ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.174181 13800 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 96ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.174595 15031 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent bootstrapping tablet: real 0.150s	user 0.074s	sys 0.005s
I20260812 06:37:01.176884 15031 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.179507 15031 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Initialized, Role: FOLLOWER
I20260812 06:37:01.180012 15031 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 18.114, Last appended by leader: 114, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.180735 15232 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:01.181166 15031 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
W20260812 06:37:01.188827 15238 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:01.189057 15238 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:01.189097 15238 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:01.189147 15238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:01.189172 15238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:01.189222 15238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:01.190709 14989 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 84ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.192615 15139 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 18 FOLLOWER]: Advancing to term 19
W20260812 06:37:01.193624 15238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:01.193768 15238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.196
W20260812 06:37:01.198987 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 21ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.199110 15238 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.196:38991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.7.212.196
--webserver_port=40525
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.15238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.196

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:01.200428 15238 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:01.201891 15238 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 06:37:01.202642 15304 mvcc.cc:204] Tried to move back new op lower bound from 7317572080033959936 to 7317572079186030592. Current Snapshot: MvccSnapshot[applied={T|T < 7317572080033959936 or (T in {7317572080033959936})}]
W20260812 06:37:01.205722 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 95ms of latency in SegmentAllocator::Sync()
E20260812 06:37:01.206259 15238 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:01.210475 15323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.210608 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.210623 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:01.210987 15238 server_base.cc:1061] running on GCE node
I20260812 06:37:01.211342 15238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:01.211860 15238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:01.213013 15238 hybrid_clock.cc:648] HybridClock initialized: now 1786516621212999 us; error 28 us; skew 500 ppm
I20260812 06:37:01.214752 15238 webserver.cc:533] Webserver started at http://127.7.212.196:40525/ using document root <none> and password file <none>
I20260812 06:37:01.215368 15238 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:01.215430 15238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:01.218797 15238 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
W20260812 06:37:01.222128 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 5ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.224273 15340 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:01.225637 15238 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.003s	sys 0.000s
I20260812 06:37:01.225731 15238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "1adfc3d9d367476090a9513b769ac76a"
format_stamp: "Formatted at 2026-08-12 06:36:52 on dist-test-slave-g170"
I20260812 06:37:01.226116 15238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:01.237563 15238 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:01.238095 15238 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:01.238242 15238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:01.238811 15238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:01.239918 15357 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:01.243112 15238 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:01.243168 15238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s	user 0.000s	sys 0.000s
I20260812 06:37:01.243219 15238 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:01.244575 15238 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:01.244617 15238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
I20260812 06:37:01.245023 15357 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap starting.
I20260812 06:37:01.255914 15238 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.196:38991
I20260812 06:37:01.255990 15543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.196:38991 every 8 connection(s)
I20260812 06:37:01.257010 15238 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 06:37:01.257668 15357 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:01.264562  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 15238
I20260812 06:37:01.264729  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 13529
I20260812 06:37:01.275339 15548 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:01.275611 15548 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:01.276260 15548 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:01.278410  8077 ts_manager.cc:194] Re-registered known tserver with Master: 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991)
I20260812 06:37:01.279428  8077 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.196:47219
W20260812 06:37:01.284394 14128 connection.cc:573] server connection from 127.7.212.197:47987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:37:01.284809  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.197:41571
--local_ip_for_outbound_sockets=127.7.212.197
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=33771
--webserver_interface=127.7.212.197
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:37:01.284906  9947 meta_cache.cc:1510] marking tablet server a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571) as failed
I20260812 06:37:01.287292  9944 meta_cache.cc:1510] marking tablet server a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571) as failed
W20260812 06:37:01.289556 11224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42250: Illegal state: replica c74db17d01e8469fadd613ce468bbc60 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.298796 14785 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.299634 14782 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.300495 14779 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.301249 14765 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.301716 14767 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.302163 14768 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.302627 14772 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.303071 14775 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57220: Illegal state: replica 9f32be8bb322423eaa290fd71a8baf10 is not leader of this config: current role FOLLOWER
W20260812 06:37:01.303511 11173 connection.cc:466] server connection from 127.7.212.197:58495 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:58495 (request call id 9) could send its response
W20260812 06:37:01.303822 14732 connection.cc:466] server connection from 127.7.212.197:40351 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.212.197:40351 (request call id 6) could send its response
I20260812 06:37:01.335994 15357 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=119 overwritten=5 applied=103 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 06:37:01.336601 15357 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Bootstrap complete.
I20260812 06:37:01.337037 15357 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent bootstrapping tablet: real 0.092s	user 0.078s	sys 0.004s
I20260812 06:37:01.339771 15357 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.342692 15357 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adfc3d9d367476090a9513b769ac76a, State: Initialized, Role: FOLLOWER
I20260812 06:37:01.343238 15357 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 18.114, Last appended by leader: 114, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.344028 15548 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:01.344237 15357 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
W20260812 06:37:01.416033 15555 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:01.416304 15555 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:01.416326 15555 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:01.416366 15555 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:01.416437 15555 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:01.416544 15555 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:01.419965 15555 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:01.420076 15555 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.197
I20260812 06:37:01.424011 15555 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.197:41571
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.7.212.197
--webserver_port=33771
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.15555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.197

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:01.425220 15555 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:01.426466 15555 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:01.429330 15555 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:01.433298 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.433319 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.433295 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:01.433662 15555 server_base.cc:1061] running on GCE node
I20260812 06:37:01.433995 15555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:01.434490 15555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:01.435657 15555 hybrid_clock.cc:648] HybridClock initialized: now 1786516621435638 us; error 29 us; skew 500 ppm
I20260812 06:37:01.437434 15555 webserver.cc:533] Webserver started at http://127.7.212.197:33771/ using document root <none> and password file <none>
I20260812 06:37:01.437951 15555 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:01.438004 15555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:01.441614 15555 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.000s	sys 0.003s
I20260812 06:37:01.443864 15587 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:01.445048 15555 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:01.445180 15555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "a60c871d36ed4e6ea4b74231f1414739"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:37:01.445590 15555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:01.489248 15252 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.489419 15252 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.490209 15252 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:01.490455 15555 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:01.490661 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 20 candidate_status { last_received { term: 19 index: 129 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:01.490736 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 20 candidate_status { last_received { term: 19 index: 129 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:01.491286 15555 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:01.491485 15555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:01.491499 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 20 candidate_status { last_received { term: 19 index: 129 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:01.492214 15555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 06:37:01.493614 14124 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Network error: Client connection negotiation failed: client connection to 127.7.212.197:41571: connect: Connection refused (error 111)
I20260812 06:37:01.493685 14124 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: d12989a6d0f445359c9b09b7ce6cb578; no voters: 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:01.493902 15252 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260812 06:37:01.494076 15606 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:01.495618 15139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 20 candidate_status { last_received { term: 19 index: 129 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:01.496937 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "d12989a6d0f445359c9b09b7ce6cb578" candidate_term: 20 candidate_status { last_received { term: 19 index: 129 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:01.497231 15472 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12989a6d0f445359c9b09b7ce6cb578 in term 18.
I20260812 06:37:01.498337 15555 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:01.498418 15555 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20260812 06:37:01.498502 15555 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:01.500028 15555 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:01.500093 15555 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:01.500458 15606 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap starting.
I20260812 06:37:01.504529 15264 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.504679 15264 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.505595 15264 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:01.505805 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 20 candidate_status { last_received { term: 19 index: 130 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:01.507239 14232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 20 candidate_status { last_received { term: 19 index: 130 } } ignore_live_leader: false dest_uuid: "d12989a6d0f445359c9b09b7ce6cb578" is_pre_election: true
I20260812 06:37:01.507267 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 20 candidate_status { last_received { term: 19 index: 130 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:01.507395 14232 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 19.
W20260812 06:37:01.510583 11172 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Network error: Client connection negotiation failed: client connection to 127.7.212.197:41571: connect: Connection refused (error 111)
I20260812 06:37:01.511618 15555 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.197:41571
I20260812 06:37:01.511698 15798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.197:41571 every 8 connection(s)
I20260812 06:37:01.513010 15139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 20 candidate_status { last_received { term: 19 index: 130 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:01.513118 15555 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 06:37:01.513453 11172 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: c74db17d01e8469fadd613ce468bbc60, d12989a6d0f445359c9b09b7ce6cb578; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739
I20260812 06:37:01.513665 15264 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260812 06:37:01.516742 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "c74db17d01e8469fadd613ce468bbc60" candidate_term: 20 candidate_status { last_received { term: 19 index: 130 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:01.516923 15472 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74db17d01e8469fadd613ce468bbc60 in term 18.
I20260812 06:37:01.515928 15606 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:01.517933  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 15555
I20260812 06:37:01.518131  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 14095
I20260812 06:37:01.523455 15306 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.523660 15306 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.525859 15306 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:01.538441 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:01.545437 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:01.545642 15472 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 18.
I20260812 06:37:01.560009 14973 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.560195 14973 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.561226 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:01.561370 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 19.
I20260812 06:37:01.565325 15265 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.565471 15265 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.567483 15265 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:37:01.570726 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:01.570825 14973 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:01.570868 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 19.
I20260812 06:37:01.572063 15013 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60; no voters: 9f32be8bb322423eaa290fd71a8baf10
I20260812 06:37:01.572453 15306 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260812 06:37:01.572523 15306 raft_consensus.cc:493] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a60c871d36ed4e6ea4b74231f1414739)
I20260812 06:37:01.572582 15306 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 19 FOLLOWER]: Advancing to term 20
I20260812 06:37:01.572288 15710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
I20260812 06:37:01.573808 15306 raft_consensus.cc:515] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.574517 15306 leader_election.cc:290] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 election: Requested vote from peers 9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921), a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571), d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139), 1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991), c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285), 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797)
I20260812 06:37:01.574755 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10"
I20260812 06:37:01.574906 14863 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 19 FOLLOWER]: Advancing to term 20
I20260812 06:37:01.575788 14863 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 20.
I20260812 06:37:01.576884 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60"
I20260812 06:37:01.577020 11308 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 19 FOLLOWER]: Advancing to term 20
W20260812 06:37:01.577705 15013 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:01.577953 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0"
I20260812 06:37:01.578087 14606 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 19 FOLLOWER]: Advancing to term 20
I20260812 06:37:01.578106 11308 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 20.
I20260812 06:37:01.579273 14606 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 20.
W20260812 06:37:01.579460 15811 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:01.528438 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.198:35139 (local address 127.7.212.195:41825)
0812 06:37:01.540312 (+ 11874us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:01.540332 (+    20us) client_negotiation.cc:175] Beginning negotiation
0812 06:37:01.540432 (+   100us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 06:37:01.578493 (+ 38061us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11769,"thread_start_us":733,"threads_started":1}
W20260812 06:37:01.579667 15013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:37:01.579840  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.198:35139
--local_ip_for_outbound_sockets=127.7.212.198
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=35371
--webserver_interface=127.7.212.198
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
I20260812 06:37:01.580054 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a"
I20260812 06:37:01.580204 15472 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 18 FOLLOWER]: Advancing to term 20
W20260812 06:37:01.582515 15831 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:01.571837 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.198:35139 (local address 127.7.212.194:33743)
0812 06:37:01.580272 (+  8435us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:01.580340 (+    68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":8367,"thread_start_us":65,"threads_started":1}
I20260812 06:37:01.582603 15472 raft_consensus.cc:2468] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in term 20.
I20260812 06:37:01.583021 15015 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, 9f32be8bb322423eaa290fd71a8baf10, c74db17d01e8469fadd613ce468bbc60; no voters: 
W20260812 06:37:01.583187 14729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection reset by peer (error 104)
I20260812 06:37:01.583684 15306 raft_consensus.cc:2804] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 20 FOLLOWER]: Leader election won for term 20
I20260812 06:37:01.583729 15710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739"
I20260812 06:37:01.583915 15306 raft_consensus.cc:697] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [term 20 LEADER]: Becoming Leader. State: Replica: 89c37a5fdd994ff58d6a23cdafbc6209, State: Running, Role: LEADER
I20260812 06:37:01.584293 15306 consensus_queue.cc:237] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 19.135, Last appended by leader: 135, Current term: 20, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:01.586679 15013 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:01.588160  8077 catalog_manager.cc:5719] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 reported cstate change: term changed from 19 to 20, leader changed from a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197) to 89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195). New cstate: current_term: 20 leader_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:01.589881 15805 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:01.590094 15805 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:01.590721 15805 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:01.592689  8072 ts_manager.cc:194] Re-registered known tserver with Master: a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571)
I20260812 06:37:01.592995 15139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 20 candidate_status { last_received { term: 19 index: 127 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
W20260812 06:37:01.595747 15820 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:01.568742 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.198:35139 (local address 127.7.212.193:58827)
0812 06:37:01.594335 (+ 25593us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:01.594404 (+    69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25528,"thread_start_us":109,"threads_started":1}
W20260812 06:37:01.595978 14463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection reset by peer (error 104)
I20260812 06:37:01.595997 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 20 candidate_status { last_received { term: 19 index: 127 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:01.596192 11308 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.597200 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 20 candidate_status { last_received { term: 19 index: 127 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:01.597425 15472 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.598182 14606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 20 candidate_status { last_received { term: 19 index: 127 } } ignore_live_leader: false dest_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" is_pre_election: true
I20260812 06:37:01.598225 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "c74db17d01e8469fadd613ce468bbc60" is_pre_election: true
I20260812 06:37:01.598400 11308 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.598405 14606 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f32be8bb322423eaa290fd71a8baf10 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.598601 14863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "9f32be8bb322423eaa290fd71a8baf10" is_pre_election: true
I20260812 06:37:01.598788 14729 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9f32be8bb322423eaa290fd71a8baf10; no voters: 1adfc3d9d367476090a9513b769ac76a, 89c37a5fdd994ff58d6a23cdafbc6209, 9e8291da04bf4d8dafd875d0dfe291f0, c74db17d01e8469fadd613ce468bbc60
I20260812 06:37:01.599267 15710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:37:01.599711 14463 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 20 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
I20260812 06:37:01.603264 15139 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" is_pre_election: true
I20260812 06:37:01.605105 15472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" candidate_term: 20 candidate_status { last_received { term: 19 index: 135 } } ignore_live_leader: false dest_uuid: "1adfc3d9d367476090a9513b769ac76a" is_pre_election: true
I20260812 06:37:01.598798 14863 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.605300 15472 raft_consensus.cc:2393] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e8291da04bf4d8dafd875d0dfe291f0 in current term 20: Already voted for candidate 89c37a5fdd994ff58d6a23cdafbc6209 in this term.
I20260812 06:37:01.605331 14973 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260812 06:37:01.605485 15013 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection refused (error 111)
I20260812 06:37:01.605621 14467 leader_election.cc:304] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9e8291da04bf4d8dafd875d0dfe291f0; no voters: 89c37a5fdd994ff58d6a23cdafbc6209, 9f32be8bb322423eaa290fd71a8baf10, a60c871d36ed4e6ea4b74231f1414739, c74db17d01e8469fadd613ce468bbc60
W20260812 06:37:01.605803 15013 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection refused (error 111)
I20260812 06:37:01.605870 15265 raft_consensus.cc:2749] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260812 06:37:01.606709 14463 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection refused (error 111)
I20260812 06:37:01.614010 15710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cd3bf4d446147de95e618239dd3074c" candidate_uuid: "9f32be8bb322423eaa290fd71a8baf10" candidate_term: 20 candidate_status { last_received { term: 19 index: 127 } } ignore_live_leader: false dest_uuid: "a60c871d36ed4e6ea4b74231f1414739" is_pre_election: true
W20260812 06:37:01.614491 14729 leader_election.cc:343] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 20 pre-election: Tablet error from VoteRequest() call to peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Illegal state: must be running to vote when last-logged opid is not known
W20260812 06:37:01.615559 14729 leader_election.cc:336] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection refused (error 111)
I20260812 06:37:01.617148  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.197:57467
I20260812 06:37:01.657017 15606 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=7 applied=119 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
I20260812 06:37:01.657590 15606 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Bootstrap complete.
W20260812 06:37:01.657773 15606 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 17ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.665517 14863 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10 [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 19 index: 127. Preceding OpId from leader: term: 20 index: 136. (index mismatch)
I20260812 06:37:01.666081 11308 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 19 index: 130. Preceding OpId from leader: term: 20 index: 136. (index mismatch)
I20260812 06:37:01.666121 15843 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 119, Time since last communication: 0.000s
I20260812 06:37:01.666850 15843 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 125, Time since last communication: 0.000s
I20260812 06:37:01.671443 14606 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0 [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 19 index: 135. Preceding OpId from leader: term: 20 index: 136. (index mismatch)
W20260812 06:37:01.671509 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.672369 15843 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 125, Time since last communication: 0.000s
W20260812 06:37:01.673763 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 152ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.674288 15013 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 -> Peer a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): Couldn't send request to peer a60c871d36ed4e6ea4b74231f1414739. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:01.675235 15606 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent bootstrapping tablet: real 0.175s	user 0.096s	sys 0.008s
I20260812 06:37:01.677091 15606 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.680011 15606 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60c871d36ed4e6ea4b74231f1414739, State: Initialized, Role: FOLLOWER
I20260812 06:37:01.680492 15606 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 119, Last appended: 19.129, Last appended by leader: 129, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.681154 15805 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:01.681435 15606 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.002s
I20260812 06:37:01.692911 15472 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 18 index: 114. Preceding OpId from leader: term: 20 index: 136. (index mismatch)
I20260812 06:37:01.693624 15306 consensus_queue.cc:1048] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 103, Time since last communication: 0.000s
W20260812 06:37:01.698068 15013 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 -> Peer d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): Couldn't send request to peer d12989a6d0f445359c9b09b7ce6cb578. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.198:35139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:01.703049 15559 mvcc.cc:204] Tried to move back new op lower bound from 7317572080002666496 to 7317572079186030592. Current Snapshot: MvccSnapshot[applied={T|T < 7317572080002666496}]
W20260812 06:37:01.717896 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 112ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.723858 13104 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 102ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.727490 15865 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 51ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.728736 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 4ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.734607 15838 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:01.734807 15838 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:01.734825 15838 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:01.734921 15838 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:01.734961 15838 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:01.735049 15838 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:01.738543 15838 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:01.738637 15838 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.198
I20260812 06:37:01.742519 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.198:35139
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.7.212.198
--webserver_port=35371
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.198

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:01.743566 15838 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:01.744697 15838 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:01.747498 15838 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:01.751152 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.751189 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:01.751122 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:01.751789 15838 server_base.cc:1061] running on GCE node
I20260812 06:37:01.752134 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:01.752632 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:01.753815 15838 hybrid_clock.cc:648] HybridClock initialized: now 1786516621753792 us; error 38 us; skew 500 ppm
I20260812 06:37:01.755604 15838 webserver.cc:533] Webserver started at http://127.7.212.198:35371/ using document root <none> and password file <none>
I20260812 06:37:01.756188 15838 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:01.756307 15838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:01.760038 15838 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:01.762094 15910 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:01.763099 15838 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:01.763231 15838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "d12989a6d0f445359c9b09b7ce6cb578"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:37:01.763630 15838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 06:37:01.781883 15865 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 7ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.783548 15710 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 19 FOLLOWER]: Advancing to term 20
I20260812 06:37:01.785574 15710 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739 [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 19 index: 129. Preceding OpId from leader: term: 19 index: 135. (index mismatch)
I20260812 06:37:01.786340 15877 consensus_queue.cc:1050] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 119, Time since last communication: 0.000s
W20260812 06:37:01.792601 15928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 22ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.795645 15838 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:01.796424 15838 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:01.796622 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:01.797312 15838 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:01.798667 15938 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:01.802171 15838 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:01.802237 15838 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
I20260812 06:37:01.802306 15838 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:01.803787 15838 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:01.803843 15838 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:01.804901 15938 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap starting.
I20260812 06:37:01.815212 15838 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.198:35139
I20260812 06:37:01.816653 15838 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 06:37:01.820113  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 15838
I20260812 06:37:01.820300  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 11140
I20260812 06:37:01.829183 15938 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Log is configured to *not* fsync() on all Append() calls
W20260812 06:37:01.830956 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 124ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.832034 16141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.198:35139 every 8 connection(s)
W20260812 06:37:01.833635 15865 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 72ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.835040 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 18ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.842808 16142 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:01.843132 16142 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:01.843843 16142 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:01.846778  8072 ts_manager.cc:194] Re-registered known tserver with Master: d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139)
I20260812 06:37:01.847518  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.198:56151
W20260812 06:37:01.848389 15813 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:01.846498 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.212.199:34285 (local address 127.7.212.195:59401)
0812 06:37:01.846685 (+   187us) negotiation.cc:107] Waiting for socket to connect
0812 06:37:01.846702 (+    17us) client_negotiation.cc:175] Beginning negotiation
0812 06:37:01.846795 (+    93us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0812 06:37:01.848246 (+  1451us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.212.199:34285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":96}
W20260812 06:37:01.848563 15015 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 -> Peer c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): Couldn't send request to peer c74db17d01e8469fadd613ce468bbc60. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.199:34285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:01.849030  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.199:34285
--local_ip_for_outbound_sockets=127.7.212.199
--tserver_master_addrs=127.7.212.254:33393
--webserver_port=34823
--webserver_interface=127.7.212.199
--builtin_ntp_servers=127.7.212.212:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false with env {}
W20260812 06:37:01.854815  9947 scanner-internal.cc:458] Time spent opening tablet: real 0.623s	user 0.003s	sys 0.000s
W20260812 06:37:01.856473  9944 scanner-internal.cc:458] Time spent opening tablet: real 0.625s	user 0.001s	sys 0.001s
W20260812 06:37:01.857034 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 88ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.858966 15928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 81ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.860658 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 43ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.937482 15938 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=7 applied=120 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20260812 06:37:01.938066 15938 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Bootstrap complete.
I20260812 06:37:01.938481 15938 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent bootstrapping tablet: real 0.135s	user 0.087s	sys 0.012s
I20260812 06:37:01.941182 15938 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:01.943738 15938 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12989a6d0f445359c9b09b7ce6cb578, State: Initialized, Role: FOLLOWER
I20260812 06:37:01.944207 15938 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 120, Last appended: 19.129, Last appended by leader: 129, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
W20260812 06:37:01.944588 15928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 43ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.945104 15938 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260812 06:37:01.945243 16142 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:01.953960 16024 raft_consensus.cc:3060] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 19 FOLLOWER]: Advancing to term 20
W20260812 06:37:01.954483 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 11ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.956344 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 12ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.956400 16024 raft_consensus.cc:1275] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578 [term 20 FOLLOWER]: Refusing update from remote peer 89c37a5fdd994ff58d6a23cdafbc6209: Log matching property violated. Preceding OpId in replica: term: 19 index: 129. Preceding OpId from leader: term: 19 index: 135. (index mismatch)
I20260812 06:37:01.957686 15877 consensus_queue.cc:1050] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 120, Time since last communication: 0.000s
W20260812 06:37:01.961678 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 57ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.968291 15865 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 1adfc3d9d367476090a9513b769ac76a: Injecting 117ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.969458 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 116ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.973366 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 99ms of latency in SegmentAllocator::Sync()
W20260812 06:37:01.976337 16171 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 9ms of latency in SegmentAllocator::Sync()
I20260812 06:37:01.979537 16154 mvcc.cc:204] Tried to move back new op lower bound from 7317572083046273024 to 7317572082016055296. Current Snapshot: MvccSnapshot[applied={T|T < 7317572080042450944 or (T in {7317572080231264256,7317572080246620160,7317572080278589440})}]
W20260812 06:37:01.996456 16171 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 21ms of latency in SegmentAllocator::Sync()
W20260812 06:37:02.004169 16153 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 06:37:02.004390 16153 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 06:37:02.004410 16153 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 06:37:02.004492 16153 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:02.004536 16153 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:02.004607 16153 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:02.008680 16153 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:02.008881 16153 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.199
I20260812 06:37:02.013005 16153 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.199:34285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.7.212.199
--webserver_port=34823
--tserver_master_addrs=127.7.212.254:33393
--never_fsync=true
--heap_profile_path=/tmp/kudu.16153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.199

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:02.014197 16153 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:02.015383 16153 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 06:37:02.018606 16153 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 06:37:02.023850 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 38ms of latency in SegmentAllocator::Sync()
W20260812 06:37:02.024658 16182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:02.024907 16177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:02.024714 16178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:02.025144 16153 server_base.cc:1061] running on GCE node
I20260812 06:37:02.025557 16153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:02.026278 16153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:02.027484 16153 hybrid_clock.cc:648] HybridClock initialized: now 1786516622027452 us; error 41 us; skew 500 ppm
I20260812 06:37:02.029736 16153 webserver.cc:533] Webserver started at http://127.7.212.199:34823/ using document root <none> and password file <none>
I20260812 06:37:02.030437 16153 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:02.030519 16153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:02.035022 16153 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:37:02.040284 16196 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:02.041379 16153 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
I20260812 06:37:02.041512 16153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "c74db17d01e8469fadd613ce468bbc60"
format_stamp: "Formatted at 2026-08-12 06:36:53 on dist-test-slave-g170"
I20260812 06:37:02.041955 16153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:02.054600 16153 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:02.055214 16153 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:02.055428 16153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:02.056011 16153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:02.057382 16208 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:02.061693 16153 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:02.061764 16153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
I20260812 06:37:02.061854 16153 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:02.063601 16153 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:02.063687 16153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 06:37:02.064014 16208 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap starting.
I20260812 06:37:02.076900 16153 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.199:34285
I20260812 06:37:02.078233 16153 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 06:37:02.079859 16208 log.cc:826] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Log is configured to *not* fsync() on all Append() calls
W20260812 06:37:02.086529 15319 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209: Injecting 48ms of latency in SegmentAllocator::Sync()
W20260812 06:37:02.087424 15026 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9e8291da04bf4d8dafd875d0dfe291f0: Injecting 86ms of latency in SegmentAllocator::Sync()
I20260812 06:37:02.089151  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 16153
I20260812 06:37:02.090325 16412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.199:34285 every 8 connection(s)
W20260812 06:37:02.090981 15928 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P a60c871d36ed4e6ea4b74231f1414739: Injecting 36ms of latency in SegmentAllocator::Sync()
W20260812 06:37:02.091650 16171 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P d12989a6d0f445359c9b09b7ce6cb578: Injecting 77ms of latency in SegmentAllocator::Sync()
W20260812 06:37:02.091686 15011 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P 9f32be8bb322423eaa290fd71a8baf10: Injecting 28ms of latency in SegmentAllocator::Sync()
I20260812 06:37:02.111858 16413 heartbeater.cc:344] Connected to a master server at 127.7.212.254:33393
I20260812 06:37:02.112165 16413 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:02.112818 16413 heartbeater.cc:507] Master 127.7.212.254:33393 requested a full tablet report, sending...
I20260812 06:37:02.114728  8072 ts_manager.cc:194] Re-registered known tserver with Master: c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285)
I20260812 06:37:02.115767  8072 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.199:33881
I20260812 06:37:02.174739 16208 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap replayed 1/1 log segments. Stats: ops{read=144 overwritten=3 applied=136 ignored=0} inserts{seen=121 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260812 06:37:02.175402 16208 tablet_bootstrap.cc:492] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Bootstrap complete.
W20260812 06:37:02.175602 16208 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 151ms of latency in SegmentAllocator::Sync()
I20260812 06:37:02.228204 15437 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.234653 15672 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.239602 14533 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.239847 14824 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.241026 16289 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.242151 15106 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260812 06:37:02.249392 16013 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 8b8d1844065f4ff88f8e66e499a4d51b | 127.7.212.254:33393 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                                Value                                                                                                 |      Tags       |         Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                                                 | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                    | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                                                 | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                                 | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                                   | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                                                  | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.7.212.212:35403 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1adfc3d9d367476090a9513b769ac76a | 127.7.212.196:38991 | HEALTHY | <none>   |       0        |       0
 89c37a5fdd994ff58d6a23cdafbc6209 | 127.7.212.195:40613 | HEALTHY | <none>   |       1        |       0
 9e8291da04bf4d8dafd875d0dfe291f0 | 127.7.212.193:37797 | HEALTHY | <none>   |       0        |       0
 9f32be8bb322423eaa290fd71a8baf10 | 127.7.212.194:34921 | HEALTHY | <none>   |       0        |       0
 a60c871d36ed4e6ea4b74231f1414739 | 127.7.212.197:41571 | HEALTHY | <none>   |       0        |       0
 c74db17d01e8469fadd613ce468bbc60 | 127.7.212.199:34285 | HEALTHY | <none>   |       0        |       0
 d12989a6d0f445359c9b09b7ce6cb578 | 127.7.212.198:35139 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
               Flag               |                                                                                              Value                                                                                               |      Tags       |      Tablet Server
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_server_key_size             | 768                                                                                                                                                                                              | experimental    | all 7 server(s) checked
 local_ip_for_outbound_sockets    | 127.7.212.193                                                                                                                                                                                    | experimental    | 127.7.212.193:37797
 local_ip_for_outbound_sockets    | 127.7.212.194                                                                                                                                                                                    | experimental    | 127.7.212.194:34921
 local_ip_for_outbound_sockets    | 127.7.212.195                                                                                                                                                                                    | experimental    | 127.7.212.195:40613
 local_ip_for_outbound_sockets    | 127.7.212.196                                                                                                                                                                                    | experimental    | 127.7.212.196:38991
 local_ip_for_outbound_sockets    | 127.7.212.197                                                                                                                                                                                    | experimental    | 127.7.212.197:41571
 local_ip_for_outbound_sockets    | 127.7.212.198                                                                                                                                                                                    | experimental    | 127.7.212.198:35139
 local_ip_for_outbound_sockets    | 127.7.212.199                                                                                                                                                                                    | experimental    | 127.7.212.199:34285
 log_inject_latency               | true                                                                                                                                                                                             | runtime,unsafe  | all 7 server(s) checked
 log_inject_latency_ms_mean       | 30                                                                                                                                                                                               | runtime,unsafe  | all 7 server(s) checked
 log_inject_latency_ms_stddev     | 60                                                                                                                                                                                               | runtime,unsafe  | all 7 server(s) checked
 never_fsync                      | true                                                                                                                                                                                             | unsafe,advanced | all 7 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                | unsafe,hidden   | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                             | unsafe          | all 7 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                               | hidden          | all 7 server(s) checked
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb | hidden          | 127.7.212.193:37797
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb | hidden          | 127.7.212.194:34921
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb | hidden          | 127.7.212.195:40613
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb | hidden          | 127.7.212.196:38991
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb | hidden          | 127.7.212.197:41571
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb | hidden          | 127.7.212.198:35139
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb | hidden          | 127.7.212.199:34285

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.7.212.212:35403 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
Tablet 2cd3bf4d446147de95e618239dd3074c of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
  9f32be8bb322423eaa290fd71a8baf10 (127.7.212.194:34921): RUNNING
  a60c871d36ed4e6ea4b74231f1414739 (127.7.212.197:41571): RUNNING
  89c37a5fdd994ff58d6a23cdafbc6209 (127.7.212.195:40613): RUNNING [LEADER]
  d12989a6d0f445359c9b09b7ce6cb578 (127.7.212.198:35139): RUNNING
  1adfc3d9d367476090a9513b769ac76a (127.7.212.196:38991): RUNNING
  c74db17d01e8469fadd613ce468bbc60 (127.7.212.199:34285): not running
    State:       BOOTSTRAPPING
    Data state:  TABLET_DATA_READY
    Last status: Bootstrap complete.
  9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): RUNNING
All reported replicas are:
  A = 9f32be8bb322423eaa290fd71a8baf10
  B = a60c871d36ed4e6ea4b74231f1414739
  C = 89c37a5fdd994ff58d6a23cdafbc6209
  D = d12989a6d0f445359c9b09b7ce6cb578
  E = 1adfc3d9d367476090a9513b769ac76a
  F = c74db17d01e8469fadd613ce468bbc60
  G = 9e8291da04bf4d8dafd875d0dfe291f0
The consensus matrix is:
 Config source |           Replicas           | Current term | Config index | Committed?
---------------+------------------------------+--------------+--------------+------------
 master        | A   B   C*  D   E   F   G    |              |              | Yes
 A             | A   B   C*  D   E   F   G    | 20           | -1           | Yes
 B             | A   B   C*  D   E   F   G    | 20           | -1           | Yes
 C             | A   B   C*  D   E   F   G    | 20           | -1           | Yes
 D             | A   B   C*  D   E   F   G    | 20           | -1           | Yes
 E             | A   B   C*  D   E   F   G    | 20           | -1           | Yes
 F             | A   B   C   D   E   F   G    | 20           | -1           | Yes
 G             | A   B   C*  D   E   F   G    | 20           | -1           | Yes

The cluster doesn't have any matching system tables
Summary by table
     Name      | RF |      Status      | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
 test-workload | 7  | UNDER_REPLICATED | 1             | 0       | 0          | 1                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 1
 Tablets        | 1
 Replicas       | 7

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy

FAILED
I20260812 06:37:02.315804  8019 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 06:37:02.327136 16208 ts_tablet_manager.cc:1403] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent bootstrapping tablet: real 0.263s	user 0.084s	sys 0.012s
I20260812 06:37:02.330004 16208 raft_consensus.cc:359] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:02.332332 16208 raft_consensus.cc:740] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74db17d01e8469fadd613ce468bbc60, State: Initialized, Role: FOLLOWER
I20260812 06:37:02.332911 16208 consensus_queue.cc:260] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 136, Last appended: 20.141, Last appended by leader: 141, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f32be8bb322423eaa290fd71a8baf10" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34921 } } peers { permanent_uuid: "a60c871d36ed4e6ea4b74231f1414739" member_type: VOTER last_known_addr { host: "127.7.212.197" port: 41571 } } peers { permanent_uuid: "89c37a5fdd994ff58d6a23cdafbc6209" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 40613 } } peers { permanent_uuid: "d12989a6d0f445359c9b09b7ce6cb578" member_type: VOTER last_known_addr { host: "127.7.212.198" port: 35139 } } peers { permanent_uuid: "1adfc3d9d367476090a9513b769ac76a" member_type: VOTER last_known_addr { host: "127.7.212.196" port: 38991 } } peers { permanent_uuid: "c74db17d01e8469fadd613ce468bbc60" member_type: VOTER last_known_addr { host: "127.7.212.199" port: 34285 } } peers { permanent_uuid: "9e8291da04bf4d8dafd875d0dfe291f0" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37797 } }
I20260812 06:37:02.333658 16413 heartbeater.cc:499] Master 127.7.212.254:33393 was elected leader, sending a full tablet report...
I20260812 06:37:02.333774 16208 ts_tablet_manager.cc:1434] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260812 06:37:02.342473 16330 raft_consensus.cc:1217] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 20.136->[20.137-20.170]   Dedup: 20.141->[20.142-20.170]
W20260812 06:37:02.350239 16492 log.cc:510] T 2cd3bf4d446147de95e618239dd3074c P c74db17d01e8469fadd613ce468bbc60: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 06:37:02.517841 15437 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.523675 14824 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.524400 16289 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.530426 15672 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.531385 14533 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 06:37:02.539925 15106 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260812 06:37:02.546666 16013 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 8b8d1844065f4ff88f8e66e499a4d51b | 127.7.212.254:33393 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                                Value                                                                                                 |      Tags       |         Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                                                 | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                    | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                                                 | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                                 | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                                   | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                                                  | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.7.212.212:35403 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 1adfc3d9d367476090a9513b769ac76a | 127.7.212.196:38991 | HEALTHY | <none>   |       0        |       0
 89c37a5fdd994ff58d6a23cdafbc6209 | 127.7.212.195:40613 | HEALTHY | <none>   |       1        |       0
 9e8291da04bf4d8dafd875d0dfe291f0 | 127.7.212.193:37797 | HEALTHY | <none>   |       0        |       0
 9f32be8bb322423eaa290fd71a8baf10 | 127.7.212.194:34921 | HEALTHY | <none>   |       0        |       0
 a60c871d36ed4e6ea4b74231f1414739 | 127.7.212.197:41571 | HEALTHY | <none>   |       0        |       0
 c74db17d01e8469fadd613ce468bbc60 | 127.7.212.199:34285 | HEALTHY | <none>   |       0        |       0
 d12989a6d0f445359c9b09b7ce6cb578 | 127.7.212.198:35139 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
               Flag               |                                                                                              Value                                                                                               |      Tags       |      Tablet Server
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_server_key_size             | 768                                                                                                                                                                                              | experimental    | all 7 server(s) checked
 local_ip_for_outbound_sockets    | 127.7.212.193                                                                                                                                                                                    | experimental    | 127.7.212.193:37797
 local_ip_for_outbound_sockets    | 127.7.212.194                                                                                                                                                                                    | experimental    | 127.7.212.194:34921
 local_ip_for_outbound_sockets    | 127.7.212.195                                                                                                                                                                                    | experimental    | 127.7.212.195:40613
 local_ip_for_outbound_sockets    | 127.7.212.196                                                                                                                                                                                    | experimental    | 127.7.212.196:38991
 local_ip_for_outbound_sockets    | 127.7.212.197                                                                                                                                                                                    | experimental    | 127.7.212.197:41571
 local_ip_for_outbound_sockets    | 127.7.212.198                                                                                                                                                                                    | experimental    | 127.7.212.198:35139
 local_ip_for_outbound_sockets    | 127.7.212.199                                                                                                                                                                                    | experimental    | 127.7.212.199:34285
 log_inject_latency               | true                                                                                                                                                                                             | runtime,unsafe  | all 7 server(s) checked
 log_inject_latency_ms_mean       | 30                                                                                                                                                                                               | runtime,unsafe  | all 7 server(s) checked
 log_inject_latency_ms_stddev     | 60                                                                                                                                                                                               | runtime,unsafe  | all 7 server(s) checked
 never_fsync                      | true                                                                                                                                                                                             | unsafe,advanced | all 7 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                | unsafe,hidden   | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                             | unsafe          | all 7 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                               | hidden          | all 7 server(s) checked
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb | hidden          | 127.7.212.193:37797
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb | hidden          | 127.7.212.194:34921
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb | hidden          | 127.7.212.195:40613
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb | hidden          | 127.7.212.196:38991
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb | hidden          | 127.7.212.197:41571
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb | hidden          | 127.7.212.198:35139
 server_dump_info_path            | /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.InsertUniqueKeysWithCrashyNodes.1786516611938192-8019-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb | hidden          | 127.7.212.199:34285

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.7.212.212:35403 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 7  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 1
 Tablets        | 1
 Replicas       | 7

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 06:37:02.578773  8019 log_verifier.cc:126] Checking tablet 2cd3bf4d446147de95e618239dd3074c
I20260812 06:37:02.660753  8019 log_verifier.cc:177] Verified matching terms for 170 ops in tablet 2cd3bf4d446147de95e618239dd3074c
I20260812 06:37:02.724011  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 14356
I20260812 06:37:02.765831  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 14693
W20260812 06:37:02.776388 15013 consensus_peers.cc:597] T 2cd3bf4d446147de95e618239dd3074c P 89c37a5fdd994ff58d6a23cdafbc6209 -> Peer 9e8291da04bf4d8dafd875d0dfe291f0 (127.7.212.193:37797): Couldn't send request to peer 9e8291da04bf4d8dafd875d0dfe291f0. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.193:37797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:02.780138  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 14965
I20260812 06:37:02.797444  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 15238
I20260812 06:37:02.810418  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 15555
I20260812 06:37:02.830094  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 15838
I20260812 06:37:02.867522  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 16153
I20260812 06:37:02.880398  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 8037
2026-08-12T06:37:02Z chronyd exiting
[       OK ] RaftConsensusITest.InsertUniqueKeysWithCrashyNodes (10961 ms)
[ RUN      ] RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue
I20260812 06:37:02.911917  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:02.911964  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:02.911976  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:02.911988  8019 ts_itest-base.cc:118] 3 replicas per TS
I20260812 06:37:02.912024  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:02Z Disabled control of system clock
I20260812 06:37:02.921483  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:42937
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:42937
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260812 06:37:03.027839 16570 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:03.028044 16570 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:03.028084 16570 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:03.031344 16570 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:03.031402 16570 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:03.031422 16570 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:03.031440 16570 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:03.035142 16570 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:42937
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:03.036155 16570 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:03.037322 16570 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:03.042658 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.042666 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.042666 16589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:03.043118 16570 server_base.cc:1061] running on GCE node
I20260812 06:37:03.043530 16570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:03.044377 16570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:03.045536 16570 hybrid_clock.cc:648] HybridClock initialized: now 1786516623045491 us; error 58 us; skew 500 ppm
I20260812 06:37:03.047266 16570 webserver.cc:533] Webserver started at http://127.7.212.254:37731/ using document root <none> and password file <none>
I20260812 06:37:03.047786 16570 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:03.047838 16570 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:03.047991 16570 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:03.049539 16570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "53e7ccd094404ec5a93f1fe524175aa3"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.050001 16570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "53e7ccd094404ec5a93f1fe524175aa3"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.053277 16570 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:03.055336 16602 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.056304 16570 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:03.056444 16570 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "53e7ccd094404ec5a93f1fe524175aa3"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.056545 16570 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:03.077811 16570 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:03.078435 16570 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:03.078629 16570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:03.086169 16683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:42937 every 8 connection(s)
I20260812 06:37:03.086179 16570 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:42937
I20260812 06:37:03.087306 16570 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:03.090140 16688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:03.095888 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3: Bootstrap starting.
I20260812 06:37:03.096674  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 16570
I20260812 06:37:03.096830  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:03.098472 16688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:03.099177 16688 log.cc:826] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:03.101020 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3: No bootstrap required, opened a new log
I20260812 06:37:03.103940 16688 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } }
I20260812 06:37:03.104173 16688 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:03.104269 16688 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e7ccd094404ec5a93f1fe524175aa3, State: Initialized, Role: FOLLOWER
I20260812 06:37:03.104661 16688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } }
I20260812 06:37:03.104827 16688 raft_consensus.cc:399] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:03.104918 16688 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:03.105051 16688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:03.105762 16688 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } }
I20260812 06:37:03.106094 16688 leader_election.cc:304] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53e7ccd094404ec5a93f1fe524175aa3; no voters: 
I20260812 06:37:03.106379 16688 leader_election.cc:290] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:03.106503 16694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:03.106738 16694 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 53e7ccd094404ec5a93f1fe524175aa3, State: Running, Role: LEADER
I20260812 06:37:03.107033 16694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } }
I20260812 06:37:03.107506 16688 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:03.108484 16697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53e7ccd094404ec5a93f1fe524175aa3. Latest consensus state: current_term: 1 leader_uuid: "53e7ccd094404ec5a93f1fe524175aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } } }
I20260812 06:37:03.108606 16697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:03.108902 16695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53e7ccd094404ec5a93f1fe524175aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7ccd094404ec5a93f1fe524175aa3" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 42937 } } }
I20260812 06:37:03.109045 16695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:03.109864 16719 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:03.112555 16719 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:03.117372 16719 catalog_manager.cc:1383] Generated new cluster ID: 16d2730263944bc4a858ca498065a166
I20260812 06:37:03.117447 16719 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:03.135658 16719 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:03.136857 16719 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:03.143185 16719 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 53e7ccd094404ec5a93f1fe524175aa3: Generated new TSK 0
I20260812 06:37:03.143800 16719 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:03.150218  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--builtin_ntp_servers=127.7.212.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260812 06:37:03.256582 16740 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:03.256846 16740 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:03.256917 16740 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:03.256995 16740 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:03.260226 16740 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:03.260380 16740 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:03.264158 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:03.265198 16740 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:03.266279 16740 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:03.272536 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.272532 16751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.272488 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:03.272946 16740 server_base.cc:1061] running on GCE node
I20260812 06:37:03.273319 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:03.273883 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:03.275069 16740 hybrid_clock.cc:648] HybridClock initialized: now 1786516623275047 us; error 38 us; skew 500 ppm
I20260812 06:37:03.276849 16740 webserver.cc:533] Webserver started at http://127.7.212.193:44099/ using document root <none> and password file <none>
I20260812 06:37:03.277407 16740 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:03.277462 16740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:03.277667 16740 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:03.279181 16740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "26fc0f31d7a6453498d965bdc22a345d"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.279665 16740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "26fc0f31d7a6453498d965bdc22a345d"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.282943 16740 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:37:03.285063 16763 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.286003 16740 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:03.286095 16740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "26fc0f31d7a6453498d965bdc22a345d"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.286211 16740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:03.314150 16740 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:03.314824 16740 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:03.315047 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:03.315651 16740 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:03.316730 16740 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:03.316804 16740 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.316877 16740 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:03.316918 16740 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.326540 16740 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:42427
I20260812 06:37:03.326570 16956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:42427 every 8 connection(s)
I20260812 06:37:03.327503 16740 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:03.336867 16960 heartbeater.cc:344] Connected to a master server at 127.7.212.254:42937
I20260812 06:37:03.337083 16960 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:03.337600 16960 heartbeater.cc:507] Master 127.7.212.254:42937 requested a full tablet report, sending...
I20260812 06:37:03.337683  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 16740
I20260812 06:37:03.337797  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:03.339146 16631 ts_manager.cc:194] Registered new tserver with Master: 26fc0f31d7a6453498d965bdc22a345d (127.7.212.193:42427)
I20260812 06:37:03.340518 16631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:41061
I20260812 06:37:03.340637  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--builtin_ntp_servers=127.7.212.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260812 06:37:03.449994 16963 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:03.450233 16963 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:03.450284 16963 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:03.450351 16963 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:03.453639 16963 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:03.453787 16963 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:03.457453 16963 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--never_fsync=true
--heap_profile_path=/tmp/kudu.16963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:03.458493 16963 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:03.459571 16963 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:03.465842 16976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.465850 16972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.465864 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:03.466219 16963 server_base.cc:1061] running on GCE node
I20260812 06:37:03.466557 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:03.467114 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:03.468295 16963 hybrid_clock.cc:648] HybridClock initialized: now 1786516623468281 us; error 33 us; skew 500 ppm
I20260812 06:37:03.470014 16963 webserver.cc:533] Webserver started at http://127.7.212.194:37017/ using document root <none> and password file <none>
I20260812 06:37:03.470557 16963 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:03.470642 16963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:03.470871 16963 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:03.472324 16963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "33a5dee2e12843c4b9725e53c2e7f305"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.472878 16963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "33a5dee2e12843c4b9725e53c2e7f305"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.476006 16963 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 06:37:03.478156 16983 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.479171 16963 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:03.479298 16963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "33a5dee2e12843c4b9725e53c2e7f305"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.479394 16963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:03.495322 16963 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:03.495926 16963 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:03.496122 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:03.496678 16963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:03.497777 16963 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:03.497851 16963 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.497920 16963 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:03.497969 16963 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.507423 16963 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:34335
I20260812 06:37:03.507627 17185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:34335 every 8 connection(s)
I20260812 06:37:03.508472 16963 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:03.517958 17186 heartbeater.cc:344] Connected to a master server at 127.7.212.254:42937
I20260812 06:37:03.517957  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 16963
I20260812 06:37:03.518121  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:03.518286 17186 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:03.518838 17186 heartbeater.cc:507] Master 127.7.212.254:42937 requested a full tablet report, sending...
I20260812 06:37:03.519976 16631 ts_manager.cc:194] Registered new tserver with Master: 33a5dee2e12843c4b9725e53c2e7f305 (127.7.212.194:34335)
I20260812 06:37:03.520336  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--builtin_ntp_servers=127.7.212.212:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260812 06:37:03.520686 16631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:39207
W20260812 06:37:03.626439 17190 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:03.626645 17190 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:03.626673 17190 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:03.626758 17190 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:03.630084 17190 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:03.630187 17190 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:03.634001 17190 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:42937
--never_fsync=true
--heap_profile_path=/tmp/kudu.17190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:03.635043 17190 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:03.636173 17190 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:03.642607 17204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.642617 17201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:03.642598 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:03.642882 17190 server_base.cc:1061] running on GCE node
I20260812 06:37:03.643363 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:03.643904 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:03.645084 17190 hybrid_clock.cc:648] HybridClock initialized: now 1786516623645063 us; error 35 us; skew 500 ppm
I20260812 06:37:03.646870 17190 webserver.cc:533] Webserver started at http://127.7.212.195:37607/ using document root <none> and password file <none>
I20260812 06:37:03.647440 17190 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:03.647495 17190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:03.647704 17190 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:03.649381 17190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8c3b1689c3374c668923424cb9416671"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.649869 17190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8c3b1689c3374c668923424cb9416671"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.653129 17190 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:03.655201 17214 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.656129 17190 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:03.656258 17190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8c3b1689c3374c668923424cb9416671"
format_stamp: "Formatted at 2026-08-12 06:37:03 on dist-test-slave-g170"
I20260812 06:37:03.656354 17190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:03.678419 17190 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:03.679117 17190 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:03.679327 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:03.679943 17190 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:03.681046 17190 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:03.681095 17190 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.681164 17190 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:03.681205 17190 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:03.691429 17190 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:37571
I20260812 06:37:03.691525 17425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:37571 every 8 connection(s)
I20260812 06:37:03.692427 17190 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:03.697395  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 17190
I20260812 06:37:03.697520  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:03.702283 17426 heartbeater.cc:344] Connected to a master server at 127.7.212.254:42937
I20260812 06:37:03.702562 17426 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:03.703120 17426 heartbeater.cc:507] Master 127.7.212.254:42937 requested a full tablet report, sending...
I20260812 06:37:03.704150 16631 ts_manager.cc:194] Registered new tserver with Master: 8c3b1689c3374c668923424cb9416671 (127.7.212.195:37571)
I20260812 06:37:03.704660 16631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:41617
I20260812 06:37:03.712332  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:03.721485 16631 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43534:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 06:37:03.723888 16631 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 06:37:03.739880 17307 tablet_service.cc:1511] Processing CreateTablet for tablet dfa85a1534874506852f6183808c9365 (DEFAULT_TABLE table=TestTable [id=243b3643ac0e41f998a5aa792524c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:03.740053 17067 tablet_service.cc:1511] Processing CreateTablet for tablet dfa85a1534874506852f6183808c9365 (DEFAULT_TABLE table=TestTable [id=243b3643ac0e41f998a5aa792524c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:03.740053 16843 tablet_service.cc:1511] Processing CreateTablet for tablet dfa85a1534874506852f6183808c9365 (DEFAULT_TABLE table=TestTable [id=243b3643ac0e41f998a5aa792524c7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:03.740916 17307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa85a1534874506852f6183808c9365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:03.741017 17067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa85a1534874506852f6183808c9365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:03.741017 16843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa85a1534874506852f6183808c9365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:03.745978 17455 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Bootstrap starting.
I20260812 06:37:03.746953 17457 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Bootstrap starting.
I20260812 06:37:03.747942 17458 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Bootstrap starting.
I20260812 06:37:03.748462 17455 tablet_bootstrap.cc:654] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:03.749114 17457 tablet_bootstrap.cc:654] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:03.749302 17455 log.cc:826] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:03.749915 17457 log.cc:826] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:03.750044 17458 tablet_bootstrap.cc:654] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:03.750777 17458 log.cc:826] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:03.751097 17455 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: No bootstrap required, opened a new log
I20260812 06:37:03.751319 17455 ts_tablet_manager.cc:1403] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:03.751754 17457 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: No bootstrap required, opened a new log
I20260812 06:37:03.751942 17457 ts_tablet_manager.cc:1403] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:03.752492 17458 tablet_bootstrap.cc:492] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: No bootstrap required, opened a new log
I20260812 06:37:03.752718 17458 ts_tablet_manager.cc:1403] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20260812 06:37:03.754151 17455 raft_consensus.cc:359] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.754365 17455 raft_consensus.cc:740] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3b1689c3374c668923424cb9416671, State: Initialized, Role: FOLLOWER
I20260812 06:37:03.754653 17455 consensus_queue.cc:260] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.755028 17457 raft_consensus.cc:359] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.755252 17457 raft_consensus.cc:740] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fc0f31d7a6453498d965bdc22a345d, State: Initialized, Role: FOLLOWER
I20260812 06:37:03.755293 17426 heartbeater.cc:499] Master 127.7.212.254:42937 was elected leader, sending a full tablet report...
I20260812 06:37:03.755519 17455 ts_tablet_manager.cc:1434] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:37:03.755527 17457 consensus_queue.cc:260] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.755692 17458 raft_consensus.cc:359] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.755897 17458 raft_consensus.cc:740] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a5dee2e12843c4b9725e53c2e7f305, State: Initialized, Role: FOLLOWER
I20260812 06:37:03.756206 16960 heartbeater.cc:499] Master 127.7.212.254:42937 was elected leader, sending a full tablet report...
I20260812 06:37:03.756155 17458 consensus_queue.cc:260] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.756860 17186 heartbeater.cc:499] Master 127.7.212.254:42937 was elected leader, sending a full tablet report...
I20260812 06:37:03.756970 17458 ts_tablet_manager.cc:1434] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
I20260812 06:37:03.757596 17457 ts_tablet_manager.cc:1434] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
W20260812 06:37:03.759389 17188 tablet.cc:2406] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:03.764612  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:03.766180  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 26fc0f31d7a6453498d965bdc22a345d to finish bootstrapping
I20260812 06:37:03.772753  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 33a5dee2e12843c4b9725e53c2e7f305 to finish bootstrapping
I20260812 06:37:03.778510  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8c3b1689c3374c668923424cb9416671 to finish bootstrapping
I20260812 06:37:03.788139 17345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dfa85a1534874506852f6183808c9365"
dest_uuid: "8c3b1689c3374c668923424cb9416671"
 from {username='slave'} at 127.0.0.1:53170
I20260812 06:37:03.788393 17345 raft_consensus.cc:493] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 06:37:03.788524 17345 raft_consensus.cc:3060] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:03.789584 17345 raft_consensus.cc:515] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.790725 17345 leader_election.cc:290] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [CANDIDATE]: Term 1 election: Requested vote from peers 33a5dee2e12843c4b9725e53c2e7f305 (127.7.212.194:34335), 26fc0f31d7a6453498d965bdc22a345d (127.7.212.193:42427)
I20260812 06:37:03.795570 17097 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa85a1534874506852f6183808c9365" candidate_uuid: "8c3b1689c3374c668923424cb9416671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "33a5dee2e12843c4b9725e53c2e7f305"
I20260812 06:37:03.795571 16872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa85a1534874506852f6183808c9365" candidate_uuid: "8c3b1689c3374c668923424cb9416671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "26fc0f31d7a6453498d965bdc22a345d"
I20260812 06:37:03.795854 16872 raft_consensus.cc:3060] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:03.795862 17097 raft_consensus.cc:3060] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:03.796905 16872 raft_consensus.cc:2468] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3b1689c3374c668923424cb9416671 in term 1.
I20260812 06:37:03.797049 17097 raft_consensus.cc:2468] T dfa85a1534874506852f6183808c9365 P 33a5dee2e12843c4b9725e53c2e7f305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3b1689c3374c668923424cb9416671 in term 1.
I20260812 06:37:03.797418 17220 leader_election.cc:304] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fc0f31d7a6453498d965bdc22a345d, 8c3b1689c3374c668923424cb9416671; no voters: 
I20260812 06:37:03.797710 17465 raft_consensus.cc:2804] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:03.798009 17465 raft_consensus.cc:697] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [term 1 LEADER]: Becoming Leader. State: Replica: 8c3b1689c3374c668923424cb9416671, State: Running, Role: LEADER
I20260812 06:37:03.798406 17465 consensus_queue.cc:237] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } }
I20260812 06:37:03.802394 16624 catalog_manager.cc:5719] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c3b1689c3374c668923424cb9416671 (127.7.212.195). New cstate: current_term: 1 leader_uuid: "8c3b1689c3374c668923424cb9416671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c3b1689c3374c668923424cb9416671" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:03.803013  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1
W20260812 06:37:03.828902 16961 tablet.cc:2406] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:03.906488  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1
W20260812 06:37:03.943444 17427 tablet.cc:2406] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:04.109620  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 0.0 1.1
I20260812 06:37:04.196594 17465 consensus_queue.cc:1048] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a5dee2e12843c4b9725e53c2e7f305" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 34335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:04.202592 17465 consensus_queue.cc:1048] T dfa85a1534874506852f6183808c9365 P 8c3b1689c3374c668923424cb9416671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26fc0f31d7a6453498d965bdc22a345d" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:04.412833  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 16963
I20260812 06:37:04.430460  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 17190
I20260812 06:37:04.440596  8019 raft_consensus-itest.cc:648] ================================== Cluster setup complete.
I20260812 06:37:04.446064 16871 raft_consensus.cc:3060] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:37:04.450565 16871 raft_consensus.cc:3060] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d [term 2 FOLLOWER]: Advancing to term 3
I20260812 06:37:04.451615 16871 pending_rounds.cc:85] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Aborting all ops after (but not including) 3
I20260812 06:37:04.451704 16871 pending_rounds.cc:107] T dfa85a1534874506852f6183808c9365 P 26fc0f31d7a6453498d965bdc22a345d: Aborting uncommitted WRITE_OP operation due to leader change: 2.4
I20260812 06:37:04.454034  8019 raft_consensus-itest.cc:251] Waiting for row count 2...
I20260812 06:37:04.461771  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 16740
I20260812 06:37:04.478205  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 16570
2026-08-12T06:37:04Z chronyd exiting
[       OK ] RaftConsensusITest.TestReplaceOperationStuckInPrepareQueue (1599 ms)
[ RUN      ] RaftConsensusITest.TestMasterNotifiedOnConfigChange
I20260812 06:37:04.511710  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:04.511759  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:04.511771  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:04.511782  8019 ts_itest-base.cc:118] 2 replicas per TS
I20260812 06:37:04.511794  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:04Z Disabled control of system clock
I20260812 06:37:04.521767  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:36907
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:36907
--master_add_server_when_underreplicated=false
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false with env {}
W20260812 06:37:04.629175 17518 flags.cc:432] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20260812 06:37:04.629391 17518 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:04.629449 17518 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:04.629539 17518 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:04.632754 17518 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:04.632822 17518 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:04.632843 17518 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:04.632859 17518 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:04.636562 17518 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:36907
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:04.637581 17518 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:04.638681 17518 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:04.643813 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:04.643813 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:04.643961 17518 server_base.cc:1061] running on GCE node
W20260812 06:37:04.644042 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:04.644510 17518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:04.645407 17518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:04.646595 17518 hybrid_clock.cc:648] HybridClock initialized: now 1786516624646584 us; error 27 us; skew 500 ppm
I20260812 06:37:04.648432 17518 webserver.cc:533] Webserver started at http://127.7.212.254:44153/ using document root <none> and password file <none>
I20260812 06:37:04.649077 17518 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:04.649161 17518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:04.649384 17518 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:04.650847 17518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "555e056bb5684aa6b2723ad7b4873b09"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.651332 17518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "555e056bb5684aa6b2723ad7b4873b09"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.654770 17518 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.002s
I20260812 06:37:04.656862 17537 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:04.657824 17518 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:04.657964 17518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "555e056bb5684aa6b2723ad7b4873b09"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.658069 17518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:04.673084 17518 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:04.673758 17518 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:04.673965 17518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:04.681602 17518 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:36907
I20260812 06:37:04.681617 17624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:36907 every 8 connection(s)
I20260812 06:37:04.682621 17518 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:04.685508 17627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:04.687429  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 17518
I20260812 06:37:04.687580  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:04.690882 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09: Bootstrap starting.
I20260812 06:37:04.693243 17627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:04.694103 17627 log.cc:826] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:04.695919 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09: No bootstrap required, opened a new log
I20260812 06:37:04.698446 17627 raft_consensus.cc:359] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } }
I20260812 06:37:04.698640 17627 raft_consensus.cc:385] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:04.698729 17627 raft_consensus.cc:740] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555e056bb5684aa6b2723ad7b4873b09, State: Initialized, Role: FOLLOWER
I20260812 06:37:04.699111 17627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } }
I20260812 06:37:04.699265 17627 raft_consensus.cc:399] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:04.699349 17627 raft_consensus.cc:493] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:04.699481 17627 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:04.700229 17627 raft_consensus.cc:515] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } }
I20260812 06:37:04.700599 17627 leader_election.cc:304] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555e056bb5684aa6b2723ad7b4873b09; no voters: 
I20260812 06:37:04.700964 17627 leader_election.cc:290] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:04.701035 17633 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:04.701217 17633 raft_consensus.cc:697] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [term 1 LEADER]: Becoming Leader. State: Replica: 555e056bb5684aa6b2723ad7b4873b09, State: Running, Role: LEADER
I20260812 06:37:04.701479 17633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } }
I20260812 06:37:04.702092 17627 sys_catalog.cc:565] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:04.702574 17638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555e056bb5684aa6b2723ad7b4873b09. Latest consensus state: current_term: 1 leader_uuid: "555e056bb5684aa6b2723ad7b4873b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } } }
I20260812 06:37:04.702800 17638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:04.702991 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555e056bb5684aa6b2723ad7b4873b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e056bb5684aa6b2723ad7b4873b09" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 36907 } } }
I20260812 06:37:04.703116 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:04.703498 17642 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:04.706525 17642 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:04.711684 17642 catalog_manager.cc:1383] Generated new cluster ID: 4dea6a33bf984941a813abf73c84f273
I20260812 06:37:04.711750 17642 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:04.728121 17642 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:04.729351 17642 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:04.739360 17642 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 555e056bb5684aa6b2723ad7b4873b09: Generated new TSK 0
I20260812 06:37:04.740092 17642 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:04.750824  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--builtin_ntp_servers=127.7.212.212:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260812 06:37:04.860569 17671 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:04.860867 17671 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:04.860975 17671 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:04.864508 17671 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:04.864624 17671 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:04.868654 17671 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--never_fsync=true
--heap_profile_path=/tmp/kudu.17671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:04.869895 17671 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:04.871078 17671 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:04.878041 17683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:04.878118 17671 server_base.cc:1061] running on GCE node
W20260812 06:37:04.878043 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:04.878043 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:04.878679 17671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:04.879278 17671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:04.880507 17671 hybrid_clock.cc:648] HybridClock initialized: now 1786516624880475 us; error 45 us; skew 500 ppm
I20260812 06:37:04.882603 17671 webserver.cc:533] Webserver started at http://127.7.212.193:37123/ using document root <none> and password file <none>
I20260812 06:37:04.883265 17671 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:04.883327 17671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:04.883536 17671 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:04.885284 17671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7ec859c647e441fbab85240288ef0bb3"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.885807 17671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7ec859c647e441fbab85240288ef0bb3"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.889539 17671 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:04.892127 17691 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:04.893412 17671 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:04.893566 17671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7ec859c647e441fbab85240288ef0bb3"
format_stamp: "Formatted at 2026-08-12 06:37:04 on dist-test-slave-g170"
I20260812 06:37:04.893733 17671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:04.905397 17671 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:04.906131 17671 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:04.906351 17671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:04.907001 17671 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:04.908128 17671 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:04.908206 17671 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:04.908288 17671 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:04.908334 17671 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:04.918721 17671 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:38541
I20260812 06:37:04.918746 17901 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:38541 every 8 connection(s)
I20260812 06:37:04.919800 17671 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:04.928181  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 17671
I20260812 06:37:04.928304  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:04.930786  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--builtin_ntp_servers=127.7.212.212:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 06:37:04.935039 17903 heartbeater.cc:344] Connected to a master server at 127.7.212.254:36907
I20260812 06:37:04.935369 17903 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:04.936136 17903 heartbeater.cc:507] Master 127.7.212.254:36907 requested a full tablet report, sending...
I20260812 06:37:04.938148 17561 ts_manager.cc:194] Registered new tserver with Master: 7ec859c647e441fbab85240288ef0bb3 (127.7.212.193:38541)
I20260812 06:37:04.940107 17561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:58267
W20260812 06:37:05.046023 17910 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:05.046241 17910 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:05.046286 17910 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:05.049707 17910 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:05.049823 17910 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:05.053828 17910 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--never_fsync=true
--heap_profile_path=/tmp/kudu.17910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:05.054960 17910 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:05.056133 17910 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:05.062857 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:05.062883 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:05.062883 17918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:05.063143 17910 server_base.cc:1061] running on GCE node
I20260812 06:37:05.063596 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:05.064162 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:05.065363 17910 hybrid_clock.cc:648] HybridClock initialized: now 1786516625065350 us; error 35 us; skew 500 ppm
I20260812 06:37:05.067194 17910 webserver.cc:533] Webserver started at http://127.7.212.194:40389/ using document root <none> and password file <none>
I20260812 06:37:05.067803 17910 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:05.067888 17910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:05.068078 17910 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:05.069674 17910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d9429734cda6463ebc79fd9e97590a21"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.070173 17910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d9429734cda6463ebc79fd9e97590a21"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.073554 17910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:05.076159 17936 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.077270 17910 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:05.077396 17910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d9429734cda6463ebc79fd9e97590a21"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.077499 17910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:05.089540 17910 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:05.090148 17910 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:05.090338 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:05.090950 17910 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:05.091928 17910 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:05.091997 17910 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.092059 17910 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:05.092104 17910 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.101814 17910 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:35107
I20260812 06:37:05.101858 18116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:35107 every 8 connection(s)
I20260812 06:37:05.102735 17910 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:05.108855  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 17910
I20260812 06:37:05.109015  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:05.111382  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--builtin_ntp_servers=127.7.212.212:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 06:37:05.113732 18117 heartbeater.cc:344] Connected to a master server at 127.7.212.254:36907
I20260812 06:37:05.113991 18117 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:05.114624 18117 heartbeater.cc:507] Master 127.7.212.254:36907 requested a full tablet report, sending...
I20260812 06:37:05.115677 17561 ts_manager.cc:194] Registered new tserver with Master: d9429734cda6463ebc79fd9e97590a21 (127.7.212.194:35107)
I20260812 06:37:05.116307 17561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:48991
W20260812 06:37:05.215518 18125 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:05.215732 18125 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:05.215808 18125 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:05.219012 18125 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:05.219161 18125 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:05.222774 18125 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:36907
--never_fsync=true
--heap_profile_path=/tmp/kudu.18125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:05.223729 18125 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:05.224817 18125 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:05.230703 18143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:05.230755 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:05.230705 18139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:05.231016 18125 server_base.cc:1061] running on GCE node
I20260812 06:37:05.231464 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:05.231994 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:05.233170 18125 hybrid_clock.cc:648] HybridClock initialized: now 1786516625233158 us; error 33 us; skew 500 ppm
I20260812 06:37:05.234758 18125 webserver.cc:533] Webserver started at http://127.7.212.195:32915/ using document root <none> and password file <none>
I20260812 06:37:05.235301 18125 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:05.235383 18125 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:05.235600 18125 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:05.237159 18125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "98a560befba74c7abc1bf8c8359d3381"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.237627 18125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "98a560befba74c7abc1bf8c8359d3381"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.240810 18125 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:05.242977 18154 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.244086 18125 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:37:05.244241 18125 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "98a560befba74c7abc1bf8c8359d3381"
format_stamp: "Formatted at 2026-08-12 06:37:05 on dist-test-slave-g170"
I20260812 06:37:05.244345 18125 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:05.259454 18125 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:05.260160 18125 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:05.260380 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:05.261065 18125 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:05.262171 18125 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:05.262259 18125 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.262321 18125 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:05.262344 18125 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:05.271960 18125 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:36551
I20260812 06:37:05.271996 18366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:36551 every 8 connection(s)
I20260812 06:37:05.272992 18125 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:05.278821  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 18125
I20260812 06:37:05.279014  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:05.282087 18367 heartbeater.cc:344] Connected to a master server at 127.7.212.254:36907
I20260812 06:37:05.282383 18367 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:05.282900 18367 heartbeater.cc:507] Master 127.7.212.254:36907 requested a full tablet report, sending...
I20260812 06:37:05.283973 17561 ts_manager.cc:194] Registered new tserver with Master: 98a560befba74c7abc1bf8c8359d3381 (127.7.212.195:36551)
I20260812 06:37:05.284526 17561 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:58801
I20260812 06:37:05.294036  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:05.303779 17561 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46488:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20260812 06:37:05.324384 17778 tablet_service.cc:1511] Processing CreateTablet for tablet 9bad55f8251b4c9a960490b53c77f0c7 (DEFAULT_TABLE table=TestTable [id=affc224025ee47b89abac986c12b9143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:05.324466 18234 tablet_service.cc:1511] Processing CreateTablet for tablet 9bad55f8251b4c9a960490b53c77f0c7 (DEFAULT_TABLE table=TestTable [id=affc224025ee47b89abac986c12b9143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:05.325448 17778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bad55f8251b4c9a960490b53c77f0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:05.325448 18234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bad55f8251b4c9a960490b53c77f0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:05.331135 18386 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Bootstrap starting.
I20260812 06:37:05.331166 18394 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Bootstrap starting.
I20260812 06:37:05.333696 18394 tablet_bootstrap.cc:654] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:05.333956 18386 tablet_bootstrap.cc:654] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:05.334496 18394 log.cc:826] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:05.334744 18386 log.cc:826] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:05.336256 18394 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: No bootstrap required, opened a new log
I20260812 06:37:05.336452 18386 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: No bootstrap required, opened a new log
I20260812 06:37:05.336462 18394 ts_tablet_manager.cc:1403] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:05.336658 18386 ts_tablet_manager.cc:1403] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 06:37:05.339389 18394 raft_consensus.cc:359] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.339572 18394 raft_consensus.cc:385] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:05.339612 18394 raft_consensus.cc:740] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec859c647e441fbab85240288ef0bb3, State: Initialized, Role: FOLLOWER
I20260812 06:37:05.339501 18386 raft_consensus.cc:359] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.339781 18386 raft_consensus.cc:385] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:05.339890 18386 raft_consensus.cc:740] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a560befba74c7abc1bf8c8359d3381, State: Initialized, Role: FOLLOWER
I20260812 06:37:05.340174 18394 consensus_queue.cc:260] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.340304 18386 consensus_queue.cc:260] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.340975 17903 heartbeater.cc:499] Master 127.7.212.254:36907 was elected leader, sending a full tablet report...
I20260812 06:37:05.341253 18394 ts_tablet_manager.cc:1434] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 06:37:05.341316 18386 ts_tablet_manager.cc:1434] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 06:37:05.341711 18367 heartbeater.cc:499] Master 127.7.212.254:36907 was elected leader, sending a full tablet report...
W20260812 06:37:05.421067 17908 tablet.cc:2406] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:05.480020 18400 raft_consensus.cc:493] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:05.480221 18400 raft_consensus.cc:515] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.481315 18400 leader_election.cc:290] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec859c647e441fbab85240288ef0bb3 (127.7.212.193:38541)
I20260812 06:37:05.485388 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bad55f8251b4c9a960490b53c77f0c7" candidate_uuid: "98a560befba74c7abc1bf8c8359d3381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec859c647e441fbab85240288ef0bb3" is_pre_election: true
I20260812 06:37:05.485739 17810 raft_consensus.cc:2468] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a560befba74c7abc1bf8c8359d3381 in term 0.
I20260812 06:37:05.486305 18161 leader_election.cc:304] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7ec859c647e441fbab85240288ef0bb3, 98a560befba74c7abc1bf8c8359d3381; no voters: 
I20260812 06:37:05.486550 18400 raft_consensus.cc:2804] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:37:05.486627 18400 raft_consensus.cc:493] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:05.486668 18400 raft_consensus.cc:3060] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:05.487596 18400 raft_consensus.cc:515] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.487893 18400 leader_election.cc:290] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [CANDIDATE]: Term 1 election: Requested vote from peers 7ec859c647e441fbab85240288ef0bb3 (127.7.212.193:38541)
I20260812 06:37:05.488303 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bad55f8251b4c9a960490b53c77f0c7" candidate_uuid: "98a560befba74c7abc1bf8c8359d3381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec859c647e441fbab85240288ef0bb3"
I20260812 06:37:05.488451 17810 raft_consensus.cc:3060] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:05.489984 17810 raft_consensus.cc:2468] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a560befba74c7abc1bf8c8359d3381 in term 1.
I20260812 06:37:05.490415 18161 leader_election.cc:304] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7ec859c647e441fbab85240288ef0bb3, 98a560befba74c7abc1bf8c8359d3381; no voters: 
I20260812 06:37:05.490636 18400 raft_consensus.cc:2804] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:05.490907 18400 raft_consensus.cc:697] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 1 LEADER]: Becoming Leader. State: Replica: 98a560befba74c7abc1bf8c8359d3381, State: Running, Role: LEADER
I20260812 06:37:05.491254 18400 consensus_queue.cc:237] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:05.494477 17558 catalog_manager.cc:5719] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a560befba74c7abc1bf8c8359d3381 (127.7.212.195). New cstate: current_term: 1 leader_uuid: "98a560befba74c7abc1bf8c8359d3381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } health_report { overall_health: HEALTHY } } }
I20260812 06:37:05.505748  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:05.507177  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7ec859c647e441fbab85240288ef0bb3 to finish bootstrapping
I20260812 06:37:05.512917  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 98a560befba74c7abc1bf8c8359d3381 to finish bootstrapping
I20260812 06:37:05.518014  8019 raft_consensus-itest.cc:1792] Finding tablet leader and waiting for things to start...
I20260812 06:37:05.518072  8019 raft_consensus-itest.cc:1812] Waiting for Master to see the current replicas...
I20260812 06:37:05.519186  8019 raft_consensus-itest.cc:1819] Tablet locations:
tablet_locations {
  tablet_id: "9bad55f8251b4c9a960490b53c77f0c7"
  DEPRECATED_stale: false
  partition {
    partition_key_start: ""
    partition_key_end: ""
  }
  interned_replicas {
    ts_info_idx: 0
    role: FOLLOWER
  }
  interned_replicas {
    ts_info_idx: 1
    role: LEADER
  }
}
ts_infos {
  permanent_uuid: "7ec859c647e441fbab85240288ef0bb3"
  rpc_addresses {
    host: "127.7.212.193"
    port: 38541
  }
}
ts_infos {
  permanent_uuid: "98a560befba74c7abc1bf8c8359d3381"
  rpc_addresses {
    host: "127.7.212.195"
    port: 36551
  }
}
W20260812 06:37:05.523851 18369 tablet.cc:2406] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:05.885970 18400 consensus_queue.cc:1048] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:05.891157  8019 raft_consensus-itest.cc:1828] Adding tserver with uuid d9429734cda6463ebc79fd9e97590a21
I20260812 06:37:05.892871 18277 consensus_queue.cc:237] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } }
I20260812 06:37:05.894361 17810 raft_consensus.cc:1275] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 1 FOLLOWER]: Refusing update from remote peer 98a560befba74c7abc1bf8c8359d3381: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:05.894953 18401 consensus_queue.cc:1048] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 06:37:05.897030 18401 raft_consensus.cc:2955] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } } }
I20260812 06:37:05.897447 17810 raft_consensus.cc:2955] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } } }
W20260812 06:37:05.898758 18158 consensus_peers.cc:597] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 -> Peer d9429734cda6463ebc79fd9e97590a21 (127.7.212.194:35107): Couldn't send request to peer d9429734cda6463ebc79fd9e97590a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bad55f8251b4c9a960490b53c77f0c7. This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:05.899950 17561 catalog_manager.cc:5719] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 reported cstate change: config changed from index -1 to 2, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) added. New cstate: current_term: 1 leader_uuid: "98a560befba74c7abc1bf8c8359d3381" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:05.902634  8019 cluster_itest_util.cc:258] Not converged past 2 yet: 1.2 <uninitialized op> 1.2
I20260812 06:37:06.006294  8019 cluster_itest_util.cc:258] Not converged past 2 yet: 1.2 <uninitialized op> 1.2
I20260812 06:37:06.118147 18117 heartbeater.cc:499] Master 127.7.212.254:36907 was elected leader, sending a full tablet report...
I20260812 06:37:06.209596  8019 cluster_itest_util.cc:258] Not converged past 2 yet: 1.2 <uninitialized op> 1.2
I20260812 06:37:06.313849 18416 ts_tablet_manager.cc:933] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Initiating tablet copy from peer 98a560befba74c7abc1bf8c8359d3381 (127.7.212.195:36551)
I20260812 06:37:06.314452 18416 tablet_copy_client.cc:323] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: tablet copy: Beginning tablet copy session from remote peer at address 127.7.212.195:36551
I20260812 06:37:06.318751 18330 tablet_copy_service.cc:140] P 98a560befba74c7abc1bf8c8359d3381: Received BeginTabletCopySession request for tablet 9bad55f8251b4c9a960490b53c77f0c7 from peer d9429734cda6463ebc79fd9e97590a21 ({username='slave'} at 127.7.212.194:39475)
I20260812 06:37:06.318897 18330 tablet_copy_service.cc:161] P 98a560befba74c7abc1bf8c8359d3381: Beginning new tablet copy session on tablet 9bad55f8251b4c9a960490b53c77f0c7 from peer d9429734cda6463ebc79fd9e97590a21 at {username='slave'} at 127.7.212.194:39475: session id = d9429734cda6463ebc79fd9e97590a21-9bad55f8251b4c9a960490b53c77f0c7
I20260812 06:37:06.320231 18330 tablet_copy_source_session.cc:215] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 06:37:06.323717 18416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bad55f8251b4c9a960490b53c77f0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:06.328333 18416 tablet_copy_client.cc:806] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: tablet copy: Starting download of 0 data blocks...
I20260812 06:37:06.328545 18416 tablet_copy_client.cc:670] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: tablet copy: Starting download of 1 WAL segments...
I20260812 06:37:06.330502 18416 tablet_copy_client.cc:538] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 06:37:06.331986 18416 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Bootstrap starting.
I20260812 06:37:06.336120 18416 log.cc:826] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:06.338423 18416 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 06:37:06.338662 18416 tablet_bootstrap.cc:492] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Bootstrap complete.
I20260812 06:37:06.338896 18416 ts_tablet_manager.cc:1403] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.006s
I20260812 06:37:06.341382 18416 raft_consensus.cc:359] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } }
I20260812 06:37:06.341576 18416 raft_consensus.cc:740] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9429734cda6463ebc79fd9e97590a21, State: Initialized, Role: FOLLOWER
I20260812 06:37:06.342000 18416 consensus_queue.cc:260] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } peers { permanent_uuid: "d9429734cda6463ebc79fd9e97590a21" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 35107 } attrs { } }
I20260812 06:37:06.342696 18416 ts_tablet_manager.cc:1434] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:37:06.343484 18330 tablet_copy_service.cc:342] P 98a560befba74c7abc1bf8c8359d3381: Request end of tablet copy session d9429734cda6463ebc79fd9e97590a21-9bad55f8251b4c9a960490b53c77f0c7 received from {username='slave'} at 127.7.212.194:39475
I20260812 06:37:06.343631 18330 tablet_copy_service.cc:434] P 98a560befba74c7abc1bf8c8359d3381: ending tablet copy session d9429734cda6463ebc79fd9e97590a21-9bad55f8251b4c9a960490b53c77f0c7 on tablet 9bad55f8251b4c9a960490b53c77f0c7 with peer d9429734cda6463ebc79fd9e97590a21
I20260812 06:37:06.512811  8019 raft_consensus-itest.cc:1835] Waiting for Master to see config change...
I20260812 06:37:06.514132  8019 raft_consensus-itest.cc:1841] Tablet locations:
tablet_locations {
  tablet_id: "9bad55f8251b4c9a960490b53c77f0c7"
  DEPRECATED_stale: false
  partition {
    partition_key_start: ""
    partition_key_end: ""
  }
  interned_replicas {
    ts_info_idx: 0
    role: FOLLOWER
  }
  interned_replicas {
    ts_info_idx: 1
    role: LEADER
  }
  interned_replicas {
    ts_info_idx: 2
    role: FOLLOWER
  }
}
ts_infos {
  permanent_uuid: "7ec859c647e441fbab85240288ef0bb3"
  rpc_addresses {
    host: "127.7.212.193"
    port: 38541
  }
}
ts_infos {
  permanent_uuid: "98a560befba74c7abc1bf8c8359d3381"
  rpc_addresses {
    host: "127.7.212.195"
    port: 36551
  }
}
ts_infos {
  permanent_uuid: "d9429734cda6463ebc79fd9e97590a21"
  rpc_addresses {
    host: "127.7.212.194"
    port: 35107
  }
}
I20260812 06:37:06.514253  8019 raft_consensus-itest.cc:1844] Removing tserver with uuid d9429734cda6463ebc79fd9e97590a21
I20260812 06:37:06.515152 18277 consensus_queue.cc:237] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } }
I20260812 06:37:06.516446 17810 raft_consensus.cc:1275] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 1 FOLLOWER]: Refusing update from remote peer 98a560befba74c7abc1bf8c8359d3381: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260812 06:37:06.516920 18400 consensus_queue.cc:1048] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260812 06:37:06.518774 18428 raft_consensus.cc:2955] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } }
I20260812 06:37:06.519258 17810 raft_consensus.cc:2955] T 9bad55f8251b4c9a960490b53c77f0c7 P 7ec859c647e441fbab85240288ef0bb3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } } }
I20260812 06:37:06.521415 17558 catalog_manager.cc:5719] T 9bad55f8251b4c9a960490b53c77f0c7 P 98a560befba74c7abc1bf8c8359d3381 reported cstate change: config changed from index 2 to 3, VOTER d9429734cda6463ebc79fd9e97590a21 (127.7.212.194) evicted. New cstate: current_term: 1 leader_uuid: "98a560befba74c7abc1bf8c8359d3381" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ec859c647e441fbab85240288ef0bb3" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a560befba74c7abc1bf8c8359d3381" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36551 } health_report { overall_health: HEALTHY } } }
I20260812 06:37:06.522677  8019 raft_consensus-itest.cc:1851] Waiting for Master to see config change...
I20260812 06:37:06.528849 18011 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bad55f8251b4c9a960490b53c77f0c7 with delete_type TABLET_DATA_TOMBSTONED (TS d9429734cda6463ebc79fd9e97590a21 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53284
I20260812 06:37:06.529364 18439 tablet_replica.cc:333] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: stopping tablet replica
I20260812 06:37:06.529699 18439 raft_consensus.cc:2243] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 06:37:06.529963 18439 raft_consensus.cc:2272] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 06:37:06.530787 18439 ts_tablet_manager.cc:1916] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 06:37:06.532979 18439 ts_tablet_manager.cc:1929] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20260812 06:37:06.533089 18439 log.cc:1199] T 9bad55f8251b4c9a960490b53c77f0c7 P d9429734cda6463ebc79fd9e97590a21: Deleting WAL directory at /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestMasterNotifiedOnConfigChange.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/9bad55f8251b4c9a960490b53c77f0c7
I20260812 06:37:06.533716 17538 catalog_manager.cc:5050] TS d9429734cda6463ebc79fd9e97590a21 (127.7.212.194:35107): tablet 9bad55f8251b4c9a960490b53c77f0c7 (table TestTable [id=affc224025ee47b89abac986c12b9143]) successfully deleted
I20260812 06:37:06.545192  8019 raft_consensus-itest.cc:1857] Tablet locations:
tablet_locations {
  tablet_id: "9bad55f8251b4c9a960490b53c77f0c7"
  DEPRECATED_stale: false
  partition {
    partition_key_start: ""
    partition_key_end: ""
  }
  interned_replicas {
    ts_info_idx: 0
    role: FOLLOWER
  }
  interned_replicas {
    ts_info_idx: 1
    role: LEADER
  }
}
ts_infos {
  permanent_uuid: "7ec859c647e441fbab85240288ef0bb3"
  rpc_addresses {
    host: "127.7.212.193"
    port: 38541
  }
}
ts_infos {
  permanent_uuid: "98a560befba74c7abc1bf8c8359d3381"
  rpc_addresses {
    host: "127.7.212.195"
    port: 36551
  }
}
I20260812 06:37:06.545670  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 17671
I20260812 06:37:06.563274  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 17910
I20260812 06:37:06.581111  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 18125
I20260812 06:37:06.598631  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 17518
2026-08-12T06:37:06Z chronyd exiting
[       OK ] RaftConsensusITest.TestMasterNotifiedOnConfigChange (2118 ms)
[ RUN      ] RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection
I20260812 06:37:06.630406  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:06.630463  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:06.630475  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:06.630486  8019 ts_itest-base.cc:118] 3 replicas per TS
I20260812 06:37:06.630514  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:06Z Disabled control of system clock
I20260812 06:37:06.640460  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:39377
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:39377 with env {}
W20260812 06:37:06.746774 18455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:06.747049 18455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:06.747128 18455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:06.750514 18455 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:06.750615 18455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:06.750654 18455 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:06.750715 18455 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:06.754565 18455 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:39377
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:39377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:06.755651 18455 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:06.756861 18455 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:06.762105 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:06.762110 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:06.762130 18463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:06.762576 18455 server_base.cc:1061] running on GCE node
I20260812 06:37:06.763048 18455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:06.763886 18455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:06.765051 18455 hybrid_clock.cc:648] HybridClock initialized: now 1786516626765028 us; error 35 us; skew 500 ppm
I20260812 06:37:06.766903 18455 webserver.cc:533] Webserver started at http://127.7.212.254:44145/ using document root <none> and password file <none>
I20260812 06:37:06.767479 18455 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:06.767539 18455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:06.767702 18455 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:06.769388 18455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "383b6feb5f3a410997ec926f48c4abeb"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:06.769855 18455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "383b6feb5f3a410997ec926f48c4abeb"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:06.773295 18455 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:37:06.775525 18478 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:06.776579 18455 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:06.776813 18455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "383b6feb5f3a410997ec926f48c4abeb"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:06.776933 18455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:06.793972 18455 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:06.794651 18455 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:06.794872 18455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:06.802408 18455 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:39377
I20260812 06:37:06.802415 18582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:39377 every 8 connection(s)
I20260812 06:37:06.803439 18455 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:06.806005  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 18455
I20260812 06:37:06.806156  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:06.806423 18584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:06.812489 18584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb: Bootstrap starting.
I20260812 06:37:06.815384 18584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:06.816202 18584 log.cc:826] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:06.817811 18584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb: No bootstrap required, opened a new log
I20260812 06:37:06.820171 18584 raft_consensus.cc:359] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } }
I20260812 06:37:06.820353 18584 raft_consensus.cc:385] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:06.820423 18584 raft_consensus.cc:740] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383b6feb5f3a410997ec926f48c4abeb, State: Initialized, Role: FOLLOWER
I20260812 06:37:06.820909 18584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } }
I20260812 06:37:06.821058 18584 raft_consensus.cc:399] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:06.821120 18584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:06.821197 18584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:06.822108 18584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } }
I20260812 06:37:06.822438 18584 leader_election.cc:304] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 383b6feb5f3a410997ec926f48c4abeb; no voters: 
I20260812 06:37:06.822706 18584 leader_election.cc:290] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:06.822943 18594 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:06.823215 18594 raft_consensus.cc:697] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [term 1 LEADER]: Becoming Leader. State: Replica: 383b6feb5f3a410997ec926f48c4abeb, State: Running, Role: LEADER
I20260812 06:37:06.823518 18594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } }
I20260812 06:37:06.823832 18584 sys_catalog.cc:565] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:06.825333 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "383b6feb5f3a410997ec926f48c4abeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } } }
I20260812 06:37:06.825307 18596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383b6feb5f3a410997ec926f48c4abeb. Latest consensus state: current_term: 1 leader_uuid: "383b6feb5f3a410997ec926f48c4abeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383b6feb5f3a410997ec926f48c4abeb" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 39377 } } }
I20260812 06:37:06.825568 18596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:06.825482 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:06.826432 18611 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:06.829676 18611 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:06.834923 18611 catalog_manager.cc:1383] Generated new cluster ID: d6f8b025efd541c3ad4acc4b640eb0ce
I20260812 06:37:06.835000 18611 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:06.847424 18611 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:06.848610 18611 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:06.858198 18611 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 383b6feb5f3a410997ec926f48c4abeb: Generated new TSK 0
I20260812 06:37:06.858868 18611 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:06.867306  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--builtin_ntp_servers=127.7.212.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000 with env {}
W20260812 06:37:06.978643 18620 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:06.978852 18620 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:06.978894 18620 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:06.982242 18620 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:06.982347 18620 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:06.986191 18620 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--never_fsync=true
--heap_profile_path=/tmp/kudu.18620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:06.987285 18620 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:06.988466 18620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:06.994728 18631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:06.994735 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:06.994735 18633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:06.995210 18620 server_base.cc:1061] running on GCE node
I20260812 06:37:06.995951 18620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:06.996482 18620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:06.997700 18620 hybrid_clock.cc:648] HybridClock initialized: now 1786516626997675 us; error 40 us; skew 500 ppm
I20260812 06:37:06.999604 18620 webserver.cc:533] Webserver started at http://127.7.212.193:38733/ using document root <none> and password file <none>
I20260812 06:37:07.000188 18620 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:07.000275 18620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:07.000499 18620 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:07.002146 18620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:07.002653 18620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:07.006167 18620 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.002s
I20260812 06:37:07.008714 18643 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.009888 18620 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:07.010048 18620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4"
format_stamp: "Formatted at 2026-08-12 06:37:06 on dist-test-slave-g170"
I20260812 06:37:07.010180 18620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:07.029194 18620 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:07.029999 18620 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:07.030249 18620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:07.030890 18620 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:07.032096 18620 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:07.032182 18620 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.032255 18620 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:07.032312 18620 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.043738 18620 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:43233
I20260812 06:37:07.043773 18852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:43233 every 8 connection(s)
I20260812 06:37:07.044803 18620 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:07.054420 18853 heartbeater.cc:344] Connected to a master server at 127.7.212.254:39377
I20260812 06:37:07.054673 18853 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:07.055050  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 18620
I20260812 06:37:07.055176  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:07.055298 18853 heartbeater.cc:507] Master 127.7.212.254:39377 requested a full tablet report, sending...
I20260812 06:37:07.056924 18519 ts_manager.cc:194] Registered new tserver with Master: 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233)
I20260812 06:37:07.057758  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--builtin_ntp_servers=127.7.212.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000 with env {}
I20260812 06:37:07.058224 18519 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:40733
W20260812 06:37:07.162607 18860 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:07.162804 18860 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:07.162845 18860 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:07.166213 18860 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:07.166318 18860 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:07.170256 18860 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--never_fsync=true
--heap_profile_path=/tmp/kudu.18860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:07.171320 18860 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:07.172462 18860 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:07.178740 18868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:07.178807 18871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:07.178838 18873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:07.179100 18860 server_base.cc:1061] running on GCE node
I20260812 06:37:07.179503 18860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:07.180011 18860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:07.181182 18860 hybrid_clock.cc:648] HybridClock initialized: now 1786516627181163 us; error 29 us; skew 500 ppm
I20260812 06:37:07.182791 18860 webserver.cc:533] Webserver started at http://127.7.212.194:42051/ using document root <none> and password file <none>
I20260812 06:37:07.183297 18860 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:07.183349 18860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:07.183498 18860 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:07.185037 18860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.185475 18860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.188555 18860 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 06:37:07.190677 18882 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.191682 18860 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:07.191776 18860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.191900 18860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:07.202425 18860 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:07.203039 18860 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:07.203238 18860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:07.203817 18860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:07.204919 18860 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:07.204972 18860 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.205036 18860 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:07.205071 18860 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.214289 18860 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:43477
I20260812 06:37:07.214308 19075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:43477 every 8 connection(s)
I20260812 06:37:07.215317 18860 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:07.224372 19076 heartbeater.cc:344] Connected to a master server at 127.7.212.254:39377
I20260812 06:37:07.224640 19076 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:07.224701  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 18860
I20260812 06:37:07.224869  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:07.225356 19076 heartbeater.cc:507] Master 127.7.212.254:39377 requested a full tablet report, sending...
I20260812 06:37:07.226472 18518 ts_manager.cc:194] Registered new tserver with Master: 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477)
I20260812 06:37:07.227190 18518 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:38333
I20260812 06:37:07.227588  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--builtin_ntp_servers=127.7.212.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000 with env {}
W20260812 06:37:07.335431 19080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:07.335675 19080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:07.335752 19080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:07.339125 19080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:07.339285 19080 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:07.343020 19080 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=50000
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:39377
--never_fsync=true
--heap_profile_path=/tmp/kudu.19080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:07.344074 19080 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:07.345278 19080 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:07.351302 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:07.351302 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:07.351292 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:07.351576 19080 server_base.cc:1061] running on GCE node
I20260812 06:37:07.351984 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:07.352546 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:07.353757 19080 hybrid_clock.cc:648] HybridClock initialized: now 1786516627353739 us; error 35 us; skew 500 ppm
I20260812 06:37:07.355383 19080 webserver.cc:533] Webserver started at http://127.7.212.195:44419/ using document root <none> and password file <none>
I20260812 06:37:07.355940 19080 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:07.356024 19080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:07.356243 19080 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:07.357800 19080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3f31e37b18e047798ec3efba01984b91"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.358310 19080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3f31e37b18e047798ec3efba01984b91"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.361617 19080 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:07.363695 19100 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.364638 19080 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:07.364813 19080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3f31e37b18e047798ec3efba01984b91"
format_stamp: "Formatted at 2026-08-12 06:37:07 on dist-test-slave-g170"
I20260812 06:37:07.364913 19080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:07.389945 19080 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:07.390507 19080 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:07.390691 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:07.391243 19080 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:07.392217 19080 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:07.392288 19080 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.392355 19080 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:07.392405 19080 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:07.401935 19080 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:39043
I20260812 06:37:07.401969 19297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:39043 every 8 connection(s)
I20260812 06:37:07.402866 19080 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:07.404721  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 19080
I20260812 06:37:07.404850  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:07.412238 19299 heartbeater.cc:344] Connected to a master server at 127.7.212.254:39377
I20260812 06:37:07.412457 19299 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:07.413033 19299 heartbeater.cc:507] Master 127.7.212.254:39377 requested a full tablet report, sending...
I20260812 06:37:07.414042 18518 ts_manager.cc:194] Registered new tserver with Master: 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043)
I20260812 06:37:07.414579 18518 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:34893
I20260812 06:37:07.419436  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:07.429006 18518 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41938:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 06:37:07.431620 18518 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 06:37:07.448745 18740 tablet_service.cc:1511] Processing CreateTablet for tablet 4359a140be5e44d6a2abc03ff77601d0 (DEFAULT_TABLE table=TestTable [id=fc47e27cbaa44cde913182be193cbdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:07.448810 18966 tablet_service.cc:1511] Processing CreateTablet for tablet 4359a140be5e44d6a2abc03ff77601d0 (DEFAULT_TABLE table=TestTable [id=fc47e27cbaa44cde913182be193cbdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:07.449013 19188 tablet_service.cc:1511] Processing CreateTablet for tablet 4359a140be5e44d6a2abc03ff77601d0 (DEFAULT_TABLE table=TestTable [id=fc47e27cbaa44cde913182be193cbdd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:07.449755 18740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4359a140be5e44d6a2abc03ff77601d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:07.449774 18966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4359a140be5e44d6a2abc03ff77601d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:07.449774 19188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4359a140be5e44d6a2abc03ff77601d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:07.454708 19331 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Bootstrap starting.
I20260812 06:37:07.455955 19335 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Bootstrap starting.
I20260812 06:37:07.455955 19332 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Bootstrap starting.
I20260812 06:37:07.457594 19331 tablet_bootstrap.cc:654] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:07.458117 19332 tablet_bootstrap.cc:654] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:07.458117 19335 tablet_bootstrap.cc:654] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:07.458385 19331 log.cc:826] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:07.458940 19335 log.cc:826] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:07.458940 19332 log.cc:826] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:07.460021 19331 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: No bootstrap required, opened a new log
I20260812 06:37:07.460217 19331 ts_tablet_manager.cc:1403] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 06:37:07.460449 19332 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: No bootstrap required, opened a new log
I20260812 06:37:07.460633 19332 ts_tablet_manager.cc:1403] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20260812 06:37:07.461791 19335 tablet_bootstrap.cc:492] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: No bootstrap required, opened a new log
I20260812 06:37:07.462026 19335 ts_tablet_manager.cc:1403] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.005s
I20260812 06:37:07.463217 19332 raft_consensus.cc:359] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.463413 19332 raft_consensus.cc:385] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:07.463349 19331 raft_consensus.cc:359] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.463478 19332 raft_consensus.cc:740] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c518d8263fe48eb948a5c0d2ff7f9c4, State: Initialized, Role: FOLLOWER
I20260812 06:37:07.463551 19331 raft_consensus.cc:385] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:07.463620 19331 raft_consensus.cc:740] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb1219db7ac4a5a9eff3fdaf78953c4, State: Initialized, Role: FOLLOWER
I20260812 06:37:07.463943 19332 consensus_queue.cc:260] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.464020 19331 consensus_queue.cc:260] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.464577 19076 heartbeater.cc:499] Master 127.7.212.254:39377 was elected leader, sending a full tablet report...
I20260812 06:37:07.464833 19332 ts_tablet_manager.cc:1434] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.001s
I20260812 06:37:07.464879 19331 ts_tablet_manager.cc:1434] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 06:37:07.464998 19335 raft_consensus.cc:359] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.465200 18853 heartbeater.cc:499] Master 127.7.212.254:39377 was elected leader, sending a full tablet report...
I20260812 06:37:07.465247 19335 raft_consensus.cc:385] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:07.465322 19335 raft_consensus.cc:740] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f31e37b18e047798ec3efba01984b91, State: Initialized, Role: FOLLOWER
W20260812 06:37:07.466192 19077 tablet.cc:2406] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:07.465996 19335 consensus_queue.cc:260] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.467206 19299 heartbeater.cc:499] Master 127.7.212.254:39377 was elected leader, sending a full tablet report...
I20260812 06:37:07.467387 19335 ts_tablet_manager.cc:1434] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:07.535535 19341 raft_consensus.cc:493] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:07.535830 19341 raft_consensus.cc:515] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.536996 19341 leader_election.cc:290] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233), 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477)
I20260812 06:37:07.541042 18776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "3f31e37b18e047798ec3efba01984b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" is_pre_election: true
I20260812 06:37:07.541373 18776 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f31e37b18e047798ec3efba01984b91 in term 0.
I20260812 06:37:07.541363 19008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "3f31e37b18e047798ec3efba01984b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" is_pre_election: true
I20260812 06:37:07.541637 19008 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f31e37b18e047798ec3efba01984b91 in term 0.
I20260812 06:37:07.541917 19105 leader_election.cc:304] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb1219db7ac4a5a9eff3fdaf78953c4, 3f31e37b18e047798ec3efba01984b91; no voters: 
I20260812 06:37:07.542194 19341 raft_consensus.cc:2804] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:37:07.542269 19341 raft_consensus.cc:493] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:07.542301 19341 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:07.543226 19341 raft_consensus.cc:515] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.543612 19341 leader_election.cc:290] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [CANDIDATE]: Term 1 election: Requested vote from peers 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233), 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477)
I20260812 06:37:07.544015 19008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "3f31e37b18e047798ec3efba01984b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4"
I20260812 06:37:07.544039 18776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "3f31e37b18e047798ec3efba01984b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4"
I20260812 06:37:07.544149 18776 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:07.544159 19008 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:07.545459 19008 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f31e37b18e047798ec3efba01984b91 in term 1.
I20260812 06:37:07.545835 19102 leader_election.cc:304] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c518d8263fe48eb948a5c0d2ff7f9c4, 3f31e37b18e047798ec3efba01984b91; no voters: 
I20260812 06:37:07.545914 18776 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f31e37b18e047798ec3efba01984b91 in term 1.
I20260812 06:37:07.546020 19341 raft_consensus.cc:2804] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 06:37:07.546229 18855 tablet.cc:2406] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:07.546257 19341 raft_consensus.cc:697] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 LEADER]: Becoming Leader. State: Replica: 3f31e37b18e047798ec3efba01984b91, State: Running, Role: LEADER
I20260812 06:37:07.546577 19341 consensus_queue.cc:237] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:07.549727 18518 catalog_manager.cc:5719] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f31e37b18e047798ec3efba01984b91 (127.7.212.195). New cstate: current_term: 1 leader_uuid: "3f31e37b18e047798ec3efba01984b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } health_report { overall_health: HEALTHY } } }
I20260812 06:37:07.562927  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:07.564489  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2eb1219db7ac4a5a9eff3fdaf78953c4 to finish bootstrapping
I20260812 06:37:07.570384  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2c518d8263fe48eb948a5c0d2ff7f9c4 to finish bootstrapping
I20260812 06:37:07.575814  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3f31e37b18e047798ec3efba01984b91 to finish bootstrapping
W20260812 06:37:07.587988 19068 generic_service.cc:139] {username='slave'} at 127.0.0.1:41236 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20260812 06:37:07.588109 19068 generic_service.cc:175] {username='slave'} at 127.0.0.1:41236 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'true'
I20260812 06:37:07.589187 19290 generic_service.cc:175] {username='slave'} at 127.0.0.1:51590 changed flags via RPC: log_inject_latency_ms_mean from '100' to '6000'
I20260812 06:37:07.590062 19290 generic_service.cc:175] {username='slave'} at 127.0.0.1:51590 changed flags via RPC: log_inject_latency from 'false' to 'true'
W20260812 06:37:07.601689 19347 log.cc:510] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Injecting 5778ms of latency in SegmentAllocator::Sync()
I20260812 06:37:07.603142 18776 raft_consensus.cc:1275] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Refusing update from remote peer 3f31e37b18e047798ec3efba01984b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 06:37:07.603241 19102 consensus_peers.cc:597] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 -> Peer 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477): Couldn't send request to peer 2c518d8263fe48eb948a5c0d2ff7f9c4. Error code: UNKNOWN_ERROR (1). Status: Illegal state: Rejected: --follower_reject_update_consensus_requests is set to true.. This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:07.603816 19341 consensus_queue.cc:1048] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 06:37:07.693359 19308 meta_cache.cc:302] tablet 4359a140be5e44d6a2abc03ff77601d0: replica 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043) has failed: Timed out: Write RPC to 127.7.212.195:39043 timed out after 0.099s (SENT)
W20260812 06:37:07.693506 19308 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 4359a140be5e44d6a2abc03ff77601d0 after 1 attempt(s): Failed to write to server: 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043): Write RPC to 127.7.212.195:39043 timed out after 0.099s (SENT)
W20260812 06:37:08.609757 18845 generic_service.cc:139] {username='slave'} at 127.0.0.1:55124 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20260812 06:37:08.609894 18845 generic_service.cc:175] {username='slave'} at 127.0.0.1:55124 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'true'
W20260812 06:37:08.616322 19105 consensus_peers.cc:597] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 -> Peer 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233): Couldn't send request to peer 2eb1219db7ac4a5a9eff3fdaf78953c4. Error code: UNKNOWN_ERROR (1). Status: Illegal state: Rejected: --follower_reject_update_consensus_requests is set to true.. This is attempt 1: this message will repeat every 5th retry.
W20260812 06:37:08.713236 19308 meta_cache.cc:302] tablet 4359a140be5e44d6a2abc03ff77601d0: replica 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043) has failed: Timed out: Write RPC to 127.7.212.195:39043 timed out after 0.099s (SENT)
W20260812 06:37:08.713416 19308 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 4359a140be5e44d6a2abc03ff77601d0 after 1 attempt(s): Failed to write to server: 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043): Write RPC to 127.7.212.195:39043 timed out after 0.099s (SENT)
I20260812 06:37:09.593679 19361 raft_consensus.cc:493] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:09.593839 19361 raft_consensus.cc:515] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:09.595103 19361 leader_election.cc:290] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233), 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043)
I20260812 06:37:09.599392 18776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" is_pre_election: true
I20260812 06:37:10.012255 19366 raft_consensus.cc:493] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3f31e37b18e047798ec3efba01984b91)
I20260812 06:37:10.012416 19366 raft_consensus.cc:515] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:10.013739 19366 leader_election.cc:290] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477), 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043)
I20260812 06:37:10.018054 19008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" is_pre_election: true
I20260812 06:37:10.018215 19008 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb1219db7ac4a5a9eff3fdaf78953c4 in term 1.
I20260812 06:37:10.018646 18645 leader_election.cc:304] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c518d8263fe48eb948a5c0d2ff7f9c4, 2eb1219db7ac4a5a9eff3fdaf78953c4; no voters: 
I20260812 06:37:10.018999 19366 raft_consensus.cc:2804] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 06:37:10.019076 19366 raft_consensus.cc:493] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3f31e37b18e047798ec3efba01984b91)
I20260812 06:37:10.019119 19366 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:37:10.020252 19366 raft_consensus.cc:515] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:10.020711 19366 leader_election.cc:290] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [CANDIDATE]: Term 2 election: Requested vote from peers 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477), 3f31e37b18e047798ec3efba01984b91 (127.7.212.195:39043)
I20260812 06:37:10.021229 19008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4"
I20260812 06:37:10.021354 19008 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:37:10.022464 19008 raft_consensus.cc:2468] T 4359a140be5e44d6a2abc03ff77601d0 P 2c518d8263fe48eb948a5c0d2ff7f9c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb1219db7ac4a5a9eff3fdaf78953c4 in term 2.
I20260812 06:37:10.022909 18645 leader_election.cc:304] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c518d8263fe48eb948a5c0d2ff7f9c4, 2eb1219db7ac4a5a9eff3fdaf78953c4; no voters: 
I20260812 06:37:10.023160 19366 raft_consensus.cc:2804] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 06:37:10.023402 19366 raft_consensus.cc:697] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [term 2 LEADER]: Becoming Leader. State: Replica: 2eb1219db7ac4a5a9eff3fdaf78953c4, State: Running, Role: LEADER
I20260812 06:37:10.023777 19366 consensus_queue.cc:237] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:10.026453 18519 catalog_manager.cc:5719] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 reported cstate change: term changed from 1 to 2, leader changed from 3f31e37b18e047798ec3efba01984b91 (127.7.212.195) to 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193). New cstate: current_term: 2 leader_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:10.034365  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 18860
W20260812 06:37:10.077621 18645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.194:43477: connect: Connection refused (error 111)
W20260812 06:37:10.078486 18645 consensus_peers.cc:597] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 -> Peer 2c518d8263fe48eb948a5c0d2ff7f9c4 (127.7.212.194:43477): Couldn't send request to peer 2c518d8263fe48eb948a5c0d2ff7f9c4. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.194:43477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 06:37:10.171464 19389 meta_cache.cc:302] tablet 4359a140be5e44d6a2abc03ff77601d0: replica 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233) has failed: Timed out: Write RPC to 127.7.212.193:43233 timed out after 0.099s (SENT)
W20260812 06:37:10.171581 19389 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 4359a140be5e44d6a2abc03ff77601d0 after 1 attempt(s): Failed to write to server: 2eb1219db7ac4a5a9eff3fdaf78953c4 (127.7.212.193:43233): Write RPC to 127.7.212.193:43233 timed out after 0.099s (SENT)
W20260812 06:37:10.173836 19102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
W20260812 06:37:10.174749 19403 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 06:37:10.173161 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.7.212.194:53667 (local address 127.7.212.195:39043)
0812 06:37:10.173850 (+   689us) server_negotiation.cc:207] Beginning negotiation
0812 06:37:10.173856 (+     6us) server_negotiation.cc:400] Waiting for connection header
0812 06:37:10.173874 (+    18us) server_negotiation.cc:408] Connection header received
0812 06:37:10.173928 (+    54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 06:37:10.173931 (+     3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 06:37:10.173981 (+    50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 06:37:10.174131 (+   150us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.212.194:53667: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":133,"threads_started":1}
I20260812 06:37:10.175362  8019 cluster_itest_util.cc:258] Not converged past 13 yet: 2.13 1.12
I20260812 06:37:10.179108 19222 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3f31e37b18e047798ec3efba01984b91" is_pre_election: true
I20260812 06:37:10.179287 19217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4359a140be5e44d6a2abc03ff77601d0" candidate_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3f31e37b18e047798ec3efba01984b91"
I20260812 06:37:10.179546 19216 raft_consensus.cc:3055] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 06:37:10.179631 19216 raft_consensus.cc:740] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3f31e37b18e047798ec3efba01984b91, State: Running, Role: LEADER
I20260812 06:37:10.179844 19216 consensus_queue.cc:260] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb1219db7ac4a5a9eff3fdaf78953c4" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 43233 } } peers { permanent_uuid: "2c518d8263fe48eb948a5c0d2ff7f9c4" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 43477 } } peers { permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 } }
I20260812 06:37:10.180181 19216 raft_consensus.cc:3060] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 1 FOLLOWER]: Advancing to term 2
I20260812 06:37:10.181295 19216 raft_consensus.cc:1275] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 2 FOLLOWER]: Refusing update from remote peer 2eb1219db7ac4a5a9eff3fdaf78953c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260812 06:37:10.181962 19366 consensus_queue.cc:1048] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f31e37b18e047798ec3efba01984b91" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 06:37:10.182096 19366 consensus_queue.cc:1243] T 4359a140be5e44d6a2abc03ff77601d0 P 2eb1219db7ac4a5a9eff3fdaf78953c4 [LEADER]: Peer 3f31e37b18e047798ec3efba01984b91 log is divergent from this leader: its last log entry 1.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260812 06:37:10.183323 19216 raft_consensus.cc:1217] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.5]   Dedup: 1.5->[]
I20260812 06:37:10.185096 19216 raft_consensus.cc:1217] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
I20260812 06:37:10.186591 19216 raft_consensus.cc:1217] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-2.13]   Dedup: 1.11->[2.12-2.13]
I20260812 06:37:10.186690 19216 pending_rounds.cc:85] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Aborting all ops after (but not including) 11
I20260812 06:37:10.186729 19216 pending_rounds.cc:107] T 4359a140be5e44d6a2abc03ff77601d0 P 3f31e37b18e047798ec3efba01984b91: Aborting uncommitted WRITE_OP operation due to leader change: 1.12
W20260812 06:37:10.186990 19216 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51602: Aborted: Op aborted by new leader
W20260812 06:37:10.187294 19216 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51602 (ReqId={client: 46fa96cdddec4009b4b19d80e43b4783, seq_no=10, attempt_no=0}) took 1573 ms (client timeout 98 ms). Trace:
W20260812 06:37:10.187353 19216 rpcz_store.cc:269] 0812 06:37:08.613991 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:37:08.614060 (+    69us) service_pool.cc:224] Handling call
0812 06:37:10.187277 (+1573217us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 06:37:08.614566 (+     0us) write_op.cc:183] PREPARE: starting on tablet 4359a140be5e44d6a2abc03ff77601d0
0812 06:37:08.614670 (+   104us) write_op.cc:432] Acquiring schema lock in shared mode
0812 06:37:08.614674 (+     4us) write_op.cc:435] Acquired schema lock
0812 06:37:08.614676 (+     2us) tablet.cc:662] Decoding operations
0812 06:37:08.614763 (+    87us) write_op.cc:620] Acquiring the partition lock for write op
0812 06:37:08.614770 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 06:37:08.614773 (+     3us) tablet.cc:685] Acquiring locks for 1 operations
0812 06:37:08.614808 (+    35us) tablet.cc:701] Row locks acquired
0812 06:37:08.614809 (+     1us) write_op.cc:260] PREPARE: finished
0812 06:37:08.614857 (+    48us) write_op.cc:270] Start()
0812 06:37:08.614884 (+    27us) write_op.cc:276] Timestamp: P: 1786516628614814 usec, L: 0
0812 06:37:08.614885 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 06:37:08.615028 (+   143us) log.cc:844] Serialized 10331 byte log entry
0812 06:37:10.186886 (+1571858us) op_driver.cc:464] REPLICATION: finished
0812 06:37:10.186900 (+    14us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0812 06:37:10.186908 (+     8us) write_op.cc:489] Releasing partition, row and schema locks
0812 06:37:10.186967 (+    59us) write_op.cc:454] Released schema lock
0812 06:37:10.186977 (+    10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"mutex_wait_us":17,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":581,"raft.queue_time_us":156,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":230,"replication_time_us":1571890}]]}
I20260812 06:37:10.278952  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 18620
I20260812 06:37:10.300089  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 19080
I20260812 06:37:10.319304  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 18455
2026-08-12T06:37:10Z chronyd exiting
[       OK ] RaftConsensusITest.TestCommitIndexFarBehindAfterLeaderElection (3726 ms)
[ RUN      ] RaftConsensusITest.TestUpdateConsensusErrorNonePrepared
I20260812 06:37:10.357311  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:10.357374  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:10.357388  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:10.357401  8019 ts_itest-base.cc:118] 3 replicas per TS
I20260812 06:37:10.357412  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:10Z Disabled control of system clock
I20260812 06:37:10.367398  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:37693
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:37693
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260812 06:37:10.473645 19427 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:10.473897 19427 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:10.473979 19427 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:10.477233 19427 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:10.477321 19427 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:10.477360 19427 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:10.477422 19427 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:10.481269 19427 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:37693
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:10.482342 19427 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:10.483500 19427 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:10.489080 19439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:10.489209 19427 server_base.cc:1061] running on GCE node
W20260812 06:37:10.489073 19442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:10.489094 19440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:10.489814 19427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:10.490747 19427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:10.491919 19427 hybrid_clock.cc:648] HybridClock initialized: now 1786516630491898 us; error 32 us; skew 500 ppm
I20260812 06:37:10.493822 19427 webserver.cc:533] Webserver started at http://127.7.212.254:33037/ using document root <none> and password file <none>
I20260812 06:37:10.494438 19427 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:10.494498 19427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:10.494773 19427 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:10.496397 19427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bc650d1f2a61403b8bd31358ead7ede2"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.496958 19427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bc650d1f2a61403b8bd31358ead7ede2"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.500368 19427 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:37:10.502720 19450 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.503756 19427 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:37:10.503891 19427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bc650d1f2a61403b8bd31358ead7ede2"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.503990 19427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:10.515815 19427 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:10.516420 19427 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:10.516613 19427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:10.524073 19538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:37693 every 8 connection(s)
I20260812 06:37:10.524080 19427 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:37693
I20260812 06:37:10.525108 19427 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:10.527848 19539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:10.532704 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: Bootstrap starting.
I20260812 06:37:10.532958  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 19427
I20260812 06:37:10.533077  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:10.535319 19539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:10.536077 19539 log.cc:826] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:10.537963 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: No bootstrap required, opened a new log
I20260812 06:37:10.540995 19539 raft_consensus.cc:359] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } }
I20260812 06:37:10.541224 19539 raft_consensus.cc:385] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:10.541289 19539 raft_consensus.cc:740] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc650d1f2a61403b8bd31358ead7ede2, State: Initialized, Role: FOLLOWER
I20260812 06:37:10.541739 19539 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } }
I20260812 06:37:10.541891 19539 raft_consensus.cc:399] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:10.541977 19539 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:10.542096 19539 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:10.543009 19539 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } }
I20260812 06:37:10.543413 19539 leader_election.cc:304] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc650d1f2a61403b8bd31358ead7ede2; no voters: 
I20260812 06:37:10.543764 19539 leader_election.cc:290] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:10.543888 19549 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:10.544111 19549 raft_consensus.cc:697] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [term 1 LEADER]: Becoming Leader. State: Replica: bc650d1f2a61403b8bd31358ead7ede2, State: Running, Role: LEADER
I20260812 06:37:10.544484 19549 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } }
I20260812 06:37:10.544945 19539 sys_catalog.cc:565] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:10.546205 19550 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc650d1f2a61403b8bd31358ead7ede2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } } }
I20260812 06:37:10.546361 19550 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:10.546205 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc650d1f2a61403b8bd31358ead7ede2. Latest consensus state: current_term: 1 leader_uuid: "bc650d1f2a61403b8bd31358ead7ede2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc650d1f2a61403b8bd31358ead7ede2" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 37693 } } }
I20260812 06:37:10.546727 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2 [sys.catalog]: This master's current role is: LEADER
W20260812 06:37:10.550016 19586 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:10.550086 19586 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:10.550168 19568 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:10.551018 19568 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:10.556802 19568 catalog_manager.cc:1383] Generated new cluster ID: f92fd823f0294f6eb3676e419ba2d408
I20260812 06:37:10.556871 19568 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:10.569579 19568 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:10.570696 19568 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:10.581291 19568 catalog_manager.cc:6092] T 00000000000000000000000000000000 P bc650d1f2a61403b8bd31358ead7ede2: Generated new TSK 0
I20260812 06:37:10.581903 19568 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:10.593233  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--builtin_ntp_servers=127.7.212.212:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260812 06:37:10.699421 19590 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:10.699678 19590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:10.699750 19590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:10.699848 19590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:10.703183 19590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:10.703349 19590 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:10.707211 19590 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--never_fsync=true
--heap_profile_path=/tmp/kudu.19590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:10.708315 19590 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:10.709498 19590 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:10.715690 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:10.715812 19602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:10.715802 19603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:10.716061 19590 server_base.cc:1061] running on GCE node
I20260812 06:37:10.716403 19590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:10.717034 19590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:10.718214 19590 hybrid_clock.cc:648] HybridClock initialized: now 1786516630718198 us; error 35 us; skew 500 ppm
I20260812 06:37:10.719874 19590 webserver.cc:533] Webserver started at http://127.7.212.193:34951/ using document root <none> and password file <none>
I20260812 06:37:10.720431 19590 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:10.720486 19590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:10.720695 19590 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:10.722242 19590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.722723 19590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.725880 19590 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:10.727924 19618 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.728899 19590 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 06:37:10.729012 19590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.729106 19590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:10.746774 19590 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:10.747397 19590 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:10.747594 19590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:10.748168 19590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:10.749236 19590 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:10.749286 19590 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.749325 19590 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:10.749339 19590 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.759238 19590 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:38777
I20260812 06:37:10.759275 19826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:38777 every 8 connection(s)
I20260812 06:37:10.760237 19590 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:10.770031 19830 heartbeater.cc:344] Connected to a master server at 127.7.212.254:37693
I20260812 06:37:10.770344 19830 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:10.770416  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 19590
I20260812 06:37:10.770543  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:10.771015 19830 heartbeater.cc:507] Master 127.7.212.254:37693 requested a full tablet report, sending...
I20260812 06:37:10.772795 19480 ts_manager.cc:194] Registered new tserver with Master: cc674ad9a5fd4d3bb6a5de9fb3e59355 (127.7.212.193:38777)
I20260812 06:37:10.773124  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--builtin_ntp_servers=127.7.212.212:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260812 06:37:10.774427 19480 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:35847
W20260812 06:37:10.884924 19836 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:10.885162 19836 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:10.885185 19836 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:10.885289 19836 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:10.888896 19836 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:10.888999 19836 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:10.892961 19836 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--never_fsync=true
--heap_profile_path=/tmp/kudu.19836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:10.894109 19836 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:10.895270 19836 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:10.901768 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:10.901721 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:10.901881 19836 server_base.cc:1061] running on GCE node
W20260812 06:37:10.901671 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:10.902384 19836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:10.902973 19836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:10.904170 19836 hybrid_clock.cc:648] HybridClock initialized: now 1786516630904146 us; error 37 us; skew 500 ppm
I20260812 06:37:10.906169 19836 webserver.cc:533] Webserver started at http://127.7.212.194:46637/ using document root <none> and password file <none>
I20260812 06:37:10.906744 19836 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:10.906839 19836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:10.907066 19836 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:10.908748 19836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "13c66286bc4b447b973db2ad3fe2ea0b"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.909298 19836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "13c66286bc4b447b973db2ad3fe2ea0b"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.912478 19836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 06:37:10.914999 19859 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.915972 19836 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:10.916074 19836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "13c66286bc4b447b973db2ad3fe2ea0b"
format_stamp: "Formatted at 2026-08-12 06:37:10 on dist-test-slave-g170"
I20260812 06:37:10.916208 19836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:10.934432 19836 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:10.935071 19836 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:10.935278 19836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:10.935858 19836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:10.936924 19836 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:10.936975 19836 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.937012 19836 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:10.937047 19836 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:10.946748 19836 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:33777
I20260812 06:37:10.946822 20059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:33777 every 8 connection(s)
I20260812 06:37:10.947731 19836 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:10.951669  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 19836
I20260812 06:37:10.951838  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:10.954556  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--builtin_ntp_servers=127.7.212.212:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260812 06:37:10.957962 20060 heartbeater.cc:344] Connected to a master server at 127.7.212.254:37693
I20260812 06:37:10.958215 20060 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:10.958765 20060 heartbeater.cc:507] Master 127.7.212.254:37693 requested a full tablet report, sending...
I20260812 06:37:10.959905 19480 ts_manager.cc:194] Registered new tserver with Master: 13c66286bc4b447b973db2ad3fe2ea0b (127.7.212.194:33777)
I20260812 06:37:10.960496 19480 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:37611
W20260812 06:37:11.064751 20065 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:11.064965 20065 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:11.064985 20065 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:11.065073 20065 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:11.068307 20065 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:11.068403 20065 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:11.072134 20065 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:37693
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:11.073122 20065 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:11.074234 20065 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:11.080827 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.080775 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.080907 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:11.081446 20065 server_base.cc:1061] running on GCE node
I20260812 06:37:11.081799 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:11.082307 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:11.083501 20065 hybrid_clock.cc:648] HybridClock initialized: now 1786516631083470 us; error 73 us; skew 500 ppm
I20260812 06:37:11.085445 20065 webserver.cc:533] Webserver started at http://127.7.212.195:43863/ using document root <none> and password file <none>
I20260812 06:37:11.086004 20065 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:11.086095 20065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:11.086315 20065 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:11.087815 20065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c7d008c843f64c86a37fbf7300748531"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.088321 20065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c7d008c843f64c86a37fbf7300748531"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.091764 20065 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:37:11.094206 20095 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.095299 20065 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:11.095450 20065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c7d008c843f64c86a37fbf7300748531"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.095554 20065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:11.130759 20065 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:11.131481 20065 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:11.131682 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:11.132313 20065 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:11.133558 20065 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:11.133632 20065 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.133699 20065 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:11.133754 20065 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.143440 20065 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:36337
I20260812 06:37:11.143481 20314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:36337 every 8 connection(s)
I20260812 06:37:11.144395 20065 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:11.157074  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 20065
I20260812 06:37:11.157223  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestUpdateConsensusErrorNonePrepared.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:11.160132 20320 heartbeater.cc:344] Connected to a master server at 127.7.212.254:37693
I20260812 06:37:11.160421 20320 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:11.161098 20320 heartbeater.cc:507] Master 127.7.212.254:37693 requested a full tablet report, sending...
I20260812 06:37:11.162349 19480 ts_manager.cc:194] Registered new tserver with Master: c7d008c843f64c86a37fbf7300748531 (127.7.212.195:36337)
I20260812 06:37:11.163029 19480 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:48503
I20260812 06:37:11.173007  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:11.182081 19480 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55302:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 06:37:11.184607 19480 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 06:37:11.201370 19935 tablet_service.cc:1511] Processing CreateTablet for tablet a11355e2caff4a788ee7ee06ae9e9e3a (DEFAULT_TABLE table=TestTable [id=2608b458e1af499c9e3f8929d3e99203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:11.201371 19704 tablet_service.cc:1511] Processing CreateTablet for tablet a11355e2caff4a788ee7ee06ae9e9e3a (DEFAULT_TABLE table=TestTable [id=2608b458e1af499c9e3f8929d3e99203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:11.202245 19704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11355e2caff4a788ee7ee06ae9e9e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:11.202246 19935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11355e2caff4a788ee7ee06ae9e9e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:11.207158 20359 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Bootstrap starting.
I20260812 06:37:11.208045 20201 tablet_service.cc:1511] Processing CreateTablet for tablet a11355e2caff4a788ee7ee06ae9e9e3a (DEFAULT_TABLE table=TestTable [id=2608b458e1af499c9e3f8929d3e99203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:11.208302 20360 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: Bootstrap starting.
I20260812 06:37:11.208978 20201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a11355e2caff4a788ee7ee06ae9e9e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:11.209867 20359 tablet_bootstrap.cc:654] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:11.210477 20360 tablet_bootstrap.cc:654] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:11.210745 20359 log.cc:826] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:11.211252 20360 log.cc:826] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:11.212505 20359 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: No bootstrap required, opened a new log
I20260812 06:37:11.212698 20359 ts_tablet_manager.cc:1403] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 06:37:11.212929 20360 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: No bootstrap required, opened a new log
I20260812 06:37:11.213111 20360 ts_tablet_manager.cc:1403] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:11.213851 20371 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: Bootstrap starting.
I20260812 06:37:11.215850 20371 tablet_bootstrap.cc:654] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:11.216035 20360 raft_consensus.cc:359] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.216045 20359 raft_consensus.cc:359] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.216254 20359 raft_consensus.cc:740] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc674ad9a5fd4d3bb6a5de9fb3e59355, State: Initialized, Role: FOLLOWER
I20260812 06:37:11.216264 20360 raft_consensus.cc:740] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c66286bc4b447b973db2ad3fe2ea0b, State: Initialized, Role: FOLLOWER
I20260812 06:37:11.216588 20371 log.cc:826] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:11.216583 20360 consensus_queue.cc:260] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.216583 20359 consensus_queue.cc:260] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.217295 19830 heartbeater.cc:499] Master 127.7.212.254:37693 was elected leader, sending a full tablet report...
I20260812 06:37:11.217454 20360 ts_tablet_manager.cc:1434] T a11355e2caff4a788ee7ee06ae9e9e3a P 13c66286bc4b447b973db2ad3fe2ea0b: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.003s
I20260812 06:37:11.217454 20359 ts_tablet_manager.cc:1434] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:11.217789 20060 heartbeater.cc:499] Master 127.7.212.254:37693 was elected leader, sending a full tablet report...
I20260812 06:37:11.219105 20371 tablet_bootstrap.cc:492] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: No bootstrap required, opened a new log
I20260812 06:37:11.219291 20371 ts_tablet_manager.cc:1403] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20260812 06:37:11.222144 20371 raft_consensus.cc:359] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.222359 20371 raft_consensus.cc:740] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d008c843f64c86a37fbf7300748531, State: Initialized, Role: FOLLOWER
I20260812 06:37:11.222695 20371 consensus_queue.cc:260] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } }
I20260812 06:37:11.223574 20371 ts_tablet_manager.cc:1434] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:37:11.223897 20320 heartbeater.cc:499] Master 127.7.212.254:37693 was elected leader, sending a full tablet report...
I20260812 06:37:11.226123  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:11.227586  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cc674ad9a5fd4d3bb6a5de9fb3e59355 to finish bootstrapping
I20260812 06:37:11.232609  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 13c66286bc4b447b973db2ad3fe2ea0b to finish bootstrapping
I20260812 06:37:11.237464  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c7d008c843f64c86a37fbf7300748531 to finish bootstrapping
I20260812 06:37:11.242107  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 19836
W20260812 06:37:11.261476 19833 tablet.cc:2406] T a11355e2caff4a788ee7ee06ae9e9e3a P cc674ad9a5fd4d3bb6a5de9fb3e59355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:11.263830  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 19590
W20260812 06:37:11.277087 20306 generic_service.cc:139] {username='slave'} at 127.0.0.1:44196 forcing change of non-runtime-safe flag follower_fail_all_prepare
I20260812 06:37:11.277251 20306 generic_service.cc:175] {username='slave'} at 127.0.0.1:44196 changed flags via RPC: follower_fail_all_prepare from 'false' to 'true'
W20260812 06:37:11.286762 20235 raft_consensus.cc:1579] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 [term 0 FOLLOWER]: Could not prepare op 'term: 0 index: 1' and following 9 ops. Status for this op: Illegal state: Rejected: --follower_fail_all_prepare is set to true.
I20260812 06:37:11.287042 20235 raft_consensus.cc:1595] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 [term 0 FOLLOWER]: Rejecting Update request from peer cc674ad9a5fd4d3bb6a5de9fb3e59355 for term 0. Could not prepare a single op due to: Illegal state: Rejected: --follower_fail_all_prepare is set to true.
I20260812 06:37:11.288434 19480 catalog_manager.cc:5719] T a11355e2caff4a788ee7ee06ae9e9e3a P c7d008c843f64c86a37fbf7300748531 reported cstate change: leader changed from <none> to cc674ad9a5fd4d3bb6a5de9fb3e59355 (127.7.212.193). New cstate: current_term: 0 leader_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7d008c843f64c86a37fbf7300748531" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 36337 } } peers { permanent_uuid: "cc674ad9a5fd4d3bb6a5de9fb3e59355" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 38777 } } peers { permanent_uuid: "13c66286bc4b447b973db2ad3fe2ea0b" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33777 } } }
I20260812 06:37:11.287839  8019 raft_consensus-itest.cc:2696] responder_uuid: "c7d008c843f64c86a37fbf7300748531" responder_term: 0 status { last_received { term: 0 index: 0 } last_committed_idx: 0 error { code: CANNOT_PREPARE status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cc674ad9a5fd4d3bb6a5de9fb3e59355 for term 0. Could not prepare a single op due to: Illegal state: Rejected: --follower_fail_all_prepare is set to true." } } last_received_current_leader { term: 0 index: 0 } }
I20260812 06:37:11.289588  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 20065
W20260812 06:37:11.307108 19456 connection.cc:573] server connection from 127.7.212.195:48503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 06:37:11.307590  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 19427
2026-08-12T06:37:11Z chronyd exiting
[       OK ] RaftConsensusITest.TestUpdateConsensusErrorNonePrepared (983 ms)
[ RUN      ] RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC
I20260812 06:37:11.341631  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:11.341682  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:11.341694  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:11.341706  8019 ts_itest-base.cc:118] 3 replicas per TS
I20260812 06:37:11.341717  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:11Z Disabled control of system clock
I20260812 06:37:11.351621  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:38029
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:38029
--catalog_manager_evict_excess_replicas=false with env {}
W20260812 06:37:11.457053 20398 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:11.457340 20398 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:11.457418 20398 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:11.460865 20398 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:11.460960 20398 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:11.460997 20398 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:11.461035 20398 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:11.464993 20398 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_evict_excess_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:38029
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:38029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:11.466146 20398 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:11.467274 20398 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:11.472693 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.472687 20409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.472724 20408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:11.472955 20398 server_base.cc:1061] running on GCE node
I20260812 06:37:11.473515 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:11.474372 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:11.475577 20398 hybrid_clock.cc:648] HybridClock initialized: now 1786516631475551 us; error 42 us; skew 500 ppm
I20260812 06:37:11.477463 20398 webserver.cc:533] Webserver started at http://127.7.212.254:41635/ using document root <none> and password file <none>
I20260812 06:37:11.478025 20398 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:11.478111 20398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:11.478313 20398 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:11.479844 20398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a00836b64cb140118b60e8b4fb30f1b1"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.480319 20398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a00836b64cb140118b60e8b4fb30f1b1"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.483546 20398 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20260812 06:37:11.485775 20421 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.486800 20398 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:11.486948 20398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a00836b64cb140118b60e8b4fb30f1b1"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.487052 20398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:11.522050 20398 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:11.522769 20398 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:11.522975 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:11.530231 20525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:38029 every 8 connection(s)
I20260812 06:37:11.530246 20398 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:38029
I20260812 06:37:11.531283 20398 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:11.534116 20527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:11.537523  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 20398
I20260812 06:37:11.537711  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:11.539880 20527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: Bootstrap starting.
I20260812 06:37:11.542042 20527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:11.542728 20527 log.cc:826] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:11.545053 20527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: No bootstrap required, opened a new log
I20260812 06:37:11.547348 20527 raft_consensus.cc:359] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } }
I20260812 06:37:11.547511 20527 raft_consensus.cc:385] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:11.547572 20527 raft_consensus.cc:740] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00836b64cb140118b60e8b4fb30f1b1, State: Initialized, Role: FOLLOWER
I20260812 06:37:11.547973 20527 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } }
I20260812 06:37:11.548086 20527 raft_consensus.cc:399] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:11.548180 20527 raft_consensus.cc:493] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:11.548306 20527 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:11.549141 20527 raft_consensus.cc:515] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } }
I20260812 06:37:11.549463 20527 leader_election.cc:304] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a00836b64cb140118b60e8b4fb30f1b1; no voters: 
I20260812 06:37:11.549751 20527 leader_election.cc:290] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:11.549950 20540 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:11.550369 20540 raft_consensus.cc:697] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [term 1 LEADER]: Becoming Leader. State: Replica: a00836b64cb140118b60e8b4fb30f1b1, State: Running, Role: LEADER
I20260812 06:37:11.550673 20540 consensus_queue.cc:237] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } }
I20260812 06:37:11.550902 20527 sys_catalog.cc:565] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:11.552218 20543 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a00836b64cb140118b60e8b4fb30f1b1. Latest consensus state: current_term: 1 leader_uuid: "a00836b64cb140118b60e8b4fb30f1b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } } }
I20260812 06:37:11.552345 20543 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:11.552686 20541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a00836b64cb140118b60e8b4fb30f1b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00836b64cb140118b60e8b4fb30f1b1" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 38029 } } }
I20260812 06:37:11.553736 20541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1 [sys.catalog]: This master's current role is: LEADER
W20260812 06:37:11.555729 20563 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:11.555815 20563 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:11.555884 20552 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:11.556627 20552 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:11.561519 20552 catalog_manager.cc:1383] Generated new cluster ID: 3bc5e46a76e04166ba4d120d6f6d099e
I20260812 06:37:11.561582 20552 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:11.588609 20552 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:11.589804 20552 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:11.600528 20552 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a00836b64cb140118b60e8b4fb30f1b1: Generated new TSK 0
I20260812 06:37:11.601254 20552 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:11.614709  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--builtin_ntp_servers=127.7.212.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--evict_failed_followers=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260812 06:37:11.727100 20567 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 06:37:11.727334 20567 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:11.727391 20567 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:11.727483 20567 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:11.730868 20567 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 06:37:11.730921 20567 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260812 06:37:11.730957 20567 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:11.731083 20567 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20260812 06:37:11.731119 20567 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:11.734864 20567 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--evict_failed_followers=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:11.735934 20567 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:11.737097 20567 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:11.743753 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.743870 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.743753 20582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:11.744210 20567 server_base.cc:1061] running on GCE node
I20260812 06:37:11.744585 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:11.745173 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:11.746359 20567 hybrid_clock.cc:648] HybridClock initialized: now 1786516631746336 us; error 43 us; skew 500 ppm
I20260812 06:37:11.748117 20567 webserver.cc:533] Webserver started at http://127.7.212.193:42627/ using document root <none> and password file <none>
I20260812 06:37:11.748684 20567 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:11.748770 20567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:11.749022 20567 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:11.750566 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a2c3fbeec1e9447b904f70020164b478"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.751060 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a2c3fbeec1e9447b904f70020164b478"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.754470 20567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.002s
I20260812 06:37:11.756559 20598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.757599 20567 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:11.757759 20567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a2c3fbeec1e9447b904f70020164b478"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.757860 20567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:11.772526 20567 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:11.773140 20567 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:11.773334 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:11.773857 20567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:11.774824 20567 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:11.774895 20567 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.774964 20567 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:11.775004 20567 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.784531 20567 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:37587
I20260812 06:37:11.784554 20810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:37587 every 8 connection(s)
I20260812 06:37:11.785502 20567 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:11.792438  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 20567
I20260812 06:37:11.792593  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:11.794940 20813 heartbeater.cc:344] Connected to a master server at 127.7.212.254:38029
I20260812 06:37:11.795220 20813 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:11.795403  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--builtin_ntp_servers=127.7.212.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--evict_failed_followers=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260812 06:37:11.795823 20813 heartbeater.cc:507] Master 127.7.212.254:38029 requested a full tablet report, sending...
I20260812 06:37:11.797402 20458 ts_manager.cc:194] Registered new tserver with Master: a2c3fbeec1e9447b904f70020164b478 (127.7.212.193:37587)
I20260812 06:37:11.798641 20458 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:39181
W20260812 06:37:11.905499 20817 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 06:37:11.905740 20817 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:11.905761 20817 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:11.905882 20817 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:11.909313 20817 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 06:37:11.909368 20817 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260812 06:37:11.909405 20817 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:11.909533 20817 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20260812 06:37:11.909566 20817 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:11.913486 20817 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--evict_failed_followers=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--never_fsync=true
--heap_profile_path=/tmp/kudu.20817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:11.914557 20817 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:11.915704 20817 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:11.922096 20831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:11.922391 20817 server_base.cc:1061] running on GCE node
W20260812 06:37:11.922462 20827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:11.922477 20826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:11.922937 20817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:11.923491 20817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:11.924685 20817 hybrid_clock.cc:648] HybridClock initialized: now 1786516631924659 us; error 48 us; skew 500 ppm
I20260812 06:37:11.926488 20817 webserver.cc:533] Webserver started at http://127.7.212.194:33467/ using document root <none> and password file <none>
I20260812 06:37:11.927079 20817 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:11.927186 20817 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:11.927421 20817 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:11.929095 20817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "44bb2a55f27949d0b00109bcb2e98718"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.929597 20817 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "44bb2a55f27949d0b00109bcb2e98718"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.933081 20817 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:11.935312 20839 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.936420 20817 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:11.936565 20817 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "44bb2a55f27949d0b00109bcb2e98718"
format_stamp: "Formatted at 2026-08-12 06:37:11 on dist-test-slave-g170"
I20260812 06:37:11.936658 20817 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:11.958886 20817 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:11.959841 20817 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:11.960055 20817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:11.960654 20817 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:11.961707 20817 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:11.961783 20817 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.961854 20817 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:11.961899 20817 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:11.972882 20817 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:46615
I20260812 06:37:11.973088 21052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:46615 every 8 connection(s)
I20260812 06:37:11.973927 20817 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:11.983443  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 20817
I20260812 06:37:11.983652  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:11.983875 21056 heartbeater.cc:344] Connected to a master server at 127.7.212.254:38029
I20260812 06:37:11.984259 21056 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:11.984957 21056 heartbeater.cc:507] Master 127.7.212.254:38029 requested a full tablet report, sending...
I20260812 06:37:11.986143 20458 ts_manager.cc:194] Registered new tserver with Master: 44bb2a55f27949d0b00109bcb2e98718 (127.7.212.194:46615)
I20260812 06:37:11.986742 20458 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:51427
I20260812 06:37:11.986756  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--builtin_ntp_servers=127.7.212.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--evict_failed_followers=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260812 06:37:12.100265 21063 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 06:37:12.100522 21063 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:12.100544 21063 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:12.100685 21063 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:12.104036 21063 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 06:37:12.104091 21063 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260812 06:37:12.104132 21063 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:12.104261 21063 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20260812 06:37:12.104297 21063 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:12.108278 21063 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--evict_failed_followers=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:38029
--never_fsync=true
--heap_profile_path=/tmp/kudu.21063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:12.109408 21063 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:12.110518 21063 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:12.116989 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:12.116989 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:12.117071 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:12.117324 21063 server_base.cc:1061] running on GCE node
I20260812 06:37:12.117908 21063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:12.118444 21063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:12.119637 21063 hybrid_clock.cc:648] HybridClock initialized: now 1786516632119616 us; error 46 us; skew 500 ppm
I20260812 06:37:12.121577 21063 webserver.cc:533] Webserver started at http://127.7.212.195:35071/ using document root <none> and password file <none>
I20260812 06:37:12.122125 21063 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:12.122184 21063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:12.122349 21063 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:12.124006 21063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3377fbe994e14f318c416467c2e731fc"
format_stamp: "Formatted at 2026-08-12 06:37:12 on dist-test-slave-g170"
I20260812 06:37:12.124447 21063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3377fbe994e14f318c416467c2e731fc"
format_stamp: "Formatted at 2026-08-12 06:37:12 on dist-test-slave-g170"
I20260812 06:37:12.127804 21063 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:37:12.130023 21085 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:12.131196 21063 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:12.131368 21063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3377fbe994e14f318c416467c2e731fc"
format_stamp: "Formatted at 2026-08-12 06:37:12 on dist-test-slave-g170"
I20260812 06:37:12.131487 21063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:12.155470 21063 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:12.156212 21063 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:12.156433 21063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:12.157136 21063 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:12.158200 21063 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:12.158275 21063 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:12.158361 21063 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:12.158403 21063 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:12.168455 21063 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:33395
I20260812 06:37:12.168480 21296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:33395 every 8 connection(s)
I20260812 06:37:12.169492 21063 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:12.174465  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 21063
I20260812 06:37:12.174585  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:12.179167 21297 heartbeater.cc:344] Connected to a master server at 127.7.212.254:38029
I20260812 06:37:12.179419 21297 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:12.179982 21297 heartbeater.cc:507] Master 127.7.212.254:38029 requested a full tablet report, sending...
I20260812 06:37:12.181041 20458 ts_manager.cc:194] Registered new tserver with Master: 3377fbe994e14f318c416467c2e731fc (127.7.212.195:33395)
I20260812 06:37:12.181540 20458 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:52009
I20260812 06:37:12.189307  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:12.198200 20458 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57520:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 06:37:12.200501 20458 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 06:37:12.216670 21180 tablet_service.cc:1511] Processing CreateTablet for tablet 9c07f5298ab540ba9620ebaf65135c3c (DEFAULT_TABLE table=TestTable [id=bc8d8d81c086437ca8c62666becc1682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:12.216676 20904 tablet_service.cc:1511] Processing CreateTablet for tablet 9c07f5298ab540ba9620ebaf65135c3c (DEFAULT_TABLE table=TestTable [id=bc8d8d81c086437ca8c62666becc1682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:12.217370 20690 tablet_service.cc:1511] Processing CreateTablet for tablet 9c07f5298ab540ba9620ebaf65135c3c (DEFAULT_TABLE table=TestTable [id=bc8d8d81c086437ca8c62666becc1682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:12.217751 20904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c07f5298ab540ba9620ebaf65135c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:12.217751 21180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c07f5298ab540ba9620ebaf65135c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:12.218220 20690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c07f5298ab540ba9620ebaf65135c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:12.222982 21332 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Bootstrap starting.
I20260812 06:37:12.223863 21335 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Bootstrap starting.
I20260812 06:37:12.225088 21334 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Bootstrap starting.
I20260812 06:37:12.225901 21332 tablet_bootstrap.cc:654] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:12.225957 21335 tablet_bootstrap.cc:654] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:12.226625 21332 log.cc:826] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:12.226727 21335 log.cc:826] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:12.227116 21334 tablet_bootstrap.cc:654] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:12.227753 21334 log.cc:826] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:12.228334 21332 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: No bootstrap required, opened a new log
I20260812 06:37:12.228541 21332 ts_tablet_manager.cc:1403] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 06:37:12.229341 21334 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: No bootstrap required, opened a new log
I20260812 06:37:12.229563 21334 ts_tablet_manager.cc:1403] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:12.229902 21335 tablet_bootstrap.cc:492] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: No bootstrap required, opened a new log
I20260812 06:37:12.230062 21335 ts_tablet_manager.cc:1403] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20260812 06:37:12.231906 21332 raft_consensus.cc:359] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.232138 21332 raft_consensus.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:12.232195 21332 raft_consensus.cc:740] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c3fbeec1e9447b904f70020164b478, State: Initialized, Role: FOLLOWER
I20260812 06:37:12.232352 21334 raft_consensus.cc:359] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.232568 21334 raft_consensus.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:12.232640 21334 raft_consensus.cc:740] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bb2a55f27949d0b00109bcb2e98718, State: Initialized, Role: FOLLOWER
I20260812 06:37:12.232638 21332 consensus_queue.cc:260] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.233090 21334 consensus_queue.cc:260] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.233129 21335 raft_consensus.cc:359] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.233326 21335 raft_consensus.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:12.233383 21335 raft_consensus.cc:740] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fbe994e14f318c416467c2e731fc, State: Initialized, Role: FOLLOWER
I20260812 06:37:12.233587 20813 heartbeater.cc:499] Master 127.7.212.254:38029 was elected leader, sending a full tablet report...
I20260812 06:37:12.233649 21332 ts_tablet_manager.cc:1434] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.003s
I20260812 06:37:12.233788 21335 consensus_queue.cc:260] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.234270 21056 heartbeater.cc:499] Master 127.7.212.254:38029 was elected leader, sending a full tablet report...
I20260812 06:37:12.234494 21297 heartbeater.cc:499] Master 127.7.212.254:38029 was elected leader, sending a full tablet report...
I20260812 06:37:12.234691 21335 ts_tablet_manager.cc:1434] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.001s
I20260812 06:37:12.235596 21334 ts_tablet_manager.cc:1434] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
W20260812 06:37:12.270481 21298 tablet.cc:2406] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:37:12.275305 21057 tablet.cc:2406] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:37:12.287271 20815 tablet.cc:2406] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:12.317137 21353 raft_consensus.cc:493] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:12.317340 21353 raft_consensus.cc:515] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.318840 21353 leader_election.cc:290] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44bb2a55f27949d0b00109bcb2e98718 (127.7.212.194:46615), a2c3fbeec1e9447b904f70020164b478 (127.7.212.193:37587)
I20260812 06:37:12.323235 20969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "3377fbe994e14f318c416467c2e731fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bb2a55f27949d0b00109bcb2e98718" is_pre_election: true
I20260812 06:37:12.323235 20723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "3377fbe994e14f318c416467c2e731fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c3fbeec1e9447b904f70020164b478" is_pre_election: true
I20260812 06:37:12.323542 20969 raft_consensus.cc:2468] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fbe994e14f318c416467c2e731fc in term 0.
I20260812 06:37:12.323542 20723 raft_consensus.cc:2468] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fbe994e14f318c416467c2e731fc in term 0.
I20260812 06:37:12.324137 21088 leader_election.cc:304] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fbe994e14f318c416467c2e731fc, a2c3fbeec1e9447b904f70020164b478; no voters: 
I20260812 06:37:12.324455 21353 raft_consensus.cc:2804] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:37:12.324553 21353 raft_consensus.cc:493] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:12.324615 21353 raft_consensus.cc:3060] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:12.325598 21353 raft_consensus.cc:515] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.325982 21353 leader_election.cc:290] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [CANDIDATE]: Term 1 election: Requested vote from peers 44bb2a55f27949d0b00109bcb2e98718 (127.7.212.194:46615), a2c3fbeec1e9447b904f70020164b478 (127.7.212.193:37587)
I20260812 06:37:12.326427 20969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "3377fbe994e14f318c416467c2e731fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bb2a55f27949d0b00109bcb2e98718"
I20260812 06:37:12.326427 20723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "3377fbe994e14f318c416467c2e731fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c3fbeec1e9447b904f70020164b478"
I20260812 06:37:12.326562 20969 raft_consensus.cc:3060] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:12.326562 20723 raft_consensus.cc:3060] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:12.327786 20969 raft_consensus.cc:2468] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fbe994e14f318c416467c2e731fc in term 1.
I20260812 06:37:12.328172 21091 leader_election.cc:304] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fbe994e14f318c416467c2e731fc, 44bb2a55f27949d0b00109bcb2e98718; no voters: 
I20260812 06:37:12.328281 20723 raft_consensus.cc:2468] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fbe994e14f318c416467c2e731fc in term 1.
I20260812 06:37:12.328377 21353 raft_consensus.cc:2804] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:12.328614 21353 raft_consensus.cc:697] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 1 LEADER]: Becoming Leader. State: Replica: 3377fbe994e14f318c416467c2e731fc, State: Running, Role: LEADER
I20260812 06:37:12.329053 21353 consensus_queue.cc:237] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:12.332597 20458 catalog_manager.cc:5719] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fbe994e14f318c416467c2e731fc (127.7.212.195). New cstate: current_term: 1 leader_uuid: "3377fbe994e14f318c416467c2e731fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } health_report { overall_health: UNKNOWN } } }
I20260812 06:37:12.364108  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:12.365857  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a2c3fbeec1e9447b904f70020164b478 to finish bootstrapping
I20260812 06:37:12.371894  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 44bb2a55f27949d0b00109bcb2e98718 to finish bootstrapping
I20260812 06:37:12.376600  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3377fbe994e14f318c416467c2e731fc to finish bootstrapping
I20260812 06:37:12.393131  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0
I20260812 06:37:12.496423  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0
I20260812 06:37:12.699615  8019 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1 0.0
I20260812 06:37:12.859748 21362 consensus_queue.cc:1048] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:12.865741 21367 consensus_queue.cc:1048] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:13.025840  8019 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20260812 06:37:13.026193  8019 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260812 06:37:13.074087 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c): 1840834 bytes log retention, and flush 1958153 bytes memory
I20260812 06:37:13.079062 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c): 1577926 bytes log retention, and flush 2998537 bytes memory
I20260812 06:37:13.141841 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.067s	user 0.028s	sys 0.007s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":228,"dirs.queue_time_us":12418,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":693,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":713,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":12,"thread_start_us":295,"threads_started":2}
I20260812 06:37:13.143118 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling LogGCOp(9c07f5298ab540ba9620ebaf65135c3c): free 920416 bytes of WAL
I20260812 06:37:13.143616 21096 log_reader.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c: removed 1 log segments from log reader
I20260812 06:37:13.143726 21096 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000001 (ops 1-8)
I20260812 06:37:13.147541 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: LogGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.004s	user 0.000s	sys 0.000s Metrics: {}
I20260812 06:37:13.148058 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c): 2761406 bytes log retention, and flush 4038921 bytes memory
I20260812 06:37:13.173237  8019 raft_consensus-itest-base.cc:274] Waiting for log GC on 3377fbe994e14f318c416467c2e731fc
I20260812 06:37:13.173480  8019 raft_consensus-itest-base.cc:279] Log GC complete on 3377fbe994e14f318c416467c2e731fc
I20260812 06:37:13.179770 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.100s	user 0.033s	sys 0.006s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":247,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2187,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":13,"thread_start_us":316,"threads_started":2}
I20260812 06:37:13.180969 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling LogGCOp(9c07f5298ab540ba9620ebaf65135c3c): free 1577926 bytes of WAL
I20260812 06:37:13.181262 20845 log_reader.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c: removed 2 log segments from log reader
I20260812 06:37:13.181337 20845 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000001 (ops 1-8)
I20260812 06:37:13.181421 20845 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000002 (ops 9-13)
I20260812 06:37:13.182222 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: LogGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 06:37:13.182518 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c): 3155672 bytes log retention, and flush 4038921 bytes memory
I20260812 06:37:13.192014 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.044s	user 0.028s	sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":20}
I20260812 06:37:13.193094 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling LogGCOp(9c07f5298ab540ba9620ebaf65135c3c): free 1840912 bytes of WAL
I20260812 06:37:13.193302 21096 log_reader.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c: removed 2 log segments from log reader
I20260812 06:37:13.193356 21096 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000002 (ops 9-15)
I20260812 06:37:13.193405 21096 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000003 (ops 16-22)
I20260812 06:37:13.194139 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: LogGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 06:37:13.194478 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling CompactRowSetsOp(9c07f5298ab540ba9620ebaf65135c3c): perf score=0.957468
I20260812 06:37:13.222533 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: FlushMRSOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.040s	user 0.037s	sys 0.001s Metrics: {"bytes_written":135937,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1246,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":25}
I20260812 06:37:13.223594 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling LogGCOp(9c07f5298ab540ba9620ebaf65135c3c): free 1577874 bytes of WAL
I20260812 06:37:13.223819 20845 log_reader.cc:385] T 9c07f5298ab540ba9620ebaf65135c3c: removed 2 log segments from log reader
I20260812 06:37:13.223874 20845 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000003 (ops 14-20)
I20260812 06:37:13.223940 20845 log.cc:1079] T 9c07f5298ab540ba9620ebaf65135c3c P 44bb2a55f27949d0b00109bcb2e98718: Deleting log segment in path: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c07f5298ab540ba9620ebaf65135c3c/wal-000000004 (ops 21-25)
I20260812 06:37:13.224720 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: LogGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 06:37:13.225076 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling CompactRowSetsOp(9c07f5298ab540ba9620ebaf65135c3c): perf score=0.957465
I20260812 06:37:13.243359 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: CompactRowSetsOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.049s	user 0.046s	sys 0.003s Metrics: {"bytes_written":135937,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264200,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":917,"lbm_reads_lt_1ms":54,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":32,"thread_start_us":343,"threads_started":5}
I20260812 06:37:13.244128 21298 maintenance_manager.cc:419] P 3377fbe994e14f318c416467c2e731fc: Scheduling UndoDeltaBlockGCOp(9c07f5298ab540ba9620ebaf65135c3c): 490 bytes on disk
I20260812 06:37:13.244617 21096 maintenance_manager.cc:643] P 3377fbe994e14f318c416467c2e731fc: UndoDeltaBlockGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260812 06:37:13.287073 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: CompactRowSetsOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.062s	user 0.061s	sys 0.000s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264478,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":683,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1279,"drs_written":1,"lbm_read_time_us":1010,"lbm_reads_lt_1ms":54,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":38,"thread_start_us":393,"threads_started":5}
I20260812 06:37:13.287678 21057 maintenance_manager.cc:419] P 44bb2a55f27949d0b00109bcb2e98718: Scheduling UndoDeltaBlockGCOp(9c07f5298ab540ba9620ebaf65135c3c): 560 bytes on disk
I20260812 06:37:13.288228 20845 maintenance_manager.cc:643] P 44bb2a55f27949d0b00109bcb2e98718: UndoDeltaBlockGCOp(9c07f5298ab540ba9620ebaf65135c3c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20260812 06:37:15.175063  8019 raft_consensus-itest-base.cc:295] Servers converged with original term 1
I20260812 06:37:15.175135  8019 raft_consensus-itest-base.cc:300] Resuming a2c3fbeec1e9447b904f70020164b478
I20260812 06:37:15.180147 21444 raft_consensus.cc:493] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3377fbe994e14f318c416467c2e731fc)
I20260812 06:37:15.180433 21444 raft_consensus.cc:515] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3377fbe994e14f318c416467c2e731fc" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 33395 } } peers { permanent_uuid: "44bb2a55f27949d0b00109bcb2e98718" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 46615 } } peers { permanent_uuid: "a2c3fbeec1e9447b904f70020164b478" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 37587 } }
I20260812 06:37:15.181753 21444 leader_election.cc:290] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3377fbe994e14f318c416467c2e731fc (127.7.212.195:33395), 44bb2a55f27949d0b00109bcb2e98718 (127.7.212.194:46615)
I20260812 06:37:15.182200 21212 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c"
dest_uuid: "3377fbe994e14f318c416467c2e731fc"
mode: GRACEFUL
new_leader_uuid: "a2c3fbeec1e9447b904f70020164b478"
 from {username='slave'} at 127.0.0.1:37788
I20260812 06:37:15.182380 21212 raft_consensus.cc:606] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [term 1 LEADER]: Received request to transfer leadership to TS a2c3fbeec1e9447b904f70020164b478
W20260812 06:37:15.185310 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
I20260812 06:37:15.186893 21212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "a2c3fbeec1e9447b904f70020164b478" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3377fbe994e14f318c416467c2e731fc" is_pre_election: true
I20260812 06:37:15.186980 20969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c07f5298ab540ba9620ebaf65135c3c" candidate_uuid: "a2c3fbeec1e9447b904f70020164b478" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44bb2a55f27949d0b00109bcb2e98718" is_pre_election: true
I20260812 06:37:15.187574 20599 leader_election.cc:304] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c3fbeec1e9447b904f70020164b478; no voters: 3377fbe994e14f318c416467c2e731fc, 44bb2a55f27949d0b00109bcb2e98718
I20260812 06:37:15.187690 21443 consensus_queue.cc:753] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [LEADER]: The logs necessary to catch up peer a2c3fbeec1e9447b904f70020164b478 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 3..33: Segment 1 which contained index 3 has been GCed)
I20260812 06:37:15.187795 21443 consensus_queue.cc:579] T 9c07f5298ab540ba9620ebaf65135c3c P 3377fbe994e14f318c416467c2e731fc [LEADER]: The logs necessary to catch up peer a2c3fbeec1e9447b904f70020164b478 have been garbage collected. The replica will never be able to catch up
I20260812 06:37:15.187916 21444 raft_consensus.cc:2749] T 9c07f5298ab540ba9620ebaf65135c3c P a2c3fbeec1e9447b904f70020164b478 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 06:37:15.189683 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.189978 20458 catalog_manager.cc:5397] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9c07f5298ab540ba9620ebaf65135c3c with cas_config_opid_index -1: no extra replica candidate found for tablet 9c07f5298ab540ba9620ebaf65135c3c (table TestTable [id=bc8d8d81c086437ca8c62666becc1682]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20260812 06:37:15.194656 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.202088 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.214021 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.233800 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.269508 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.337802 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.469736 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:15.730050 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
W20260812 06:37:16.245996 21130 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37788: Service unavailable: leader transfer in progress
I20260812 06:37:17.255136  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 20567
I20260812 06:37:17.270723  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 20817
I20260812 06:37:17.289073  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 21063
I20260812 06:37:17.302588  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 20398
2026-08-12T06:37:17Z chronyd exiting
[       OK ] RaftConsensusITest.TestLeaderTransferWhenFollowerFallsBehindLeaderGC (5995 ms)
[----------] 6 tests from RaftConsensusITest (25386 ms total)

[----------] 1 test from RaftConsensusParamReplicationModesITest
[ RUN      ] RaftConsensusParamReplicationModesITest.Test_KUDU_1735/1
I20260812 06:37:17.337301  8019 ts_itest-base.cc:115] Starting cluster with:
I20260812 06:37:17.337357  8019 ts_itest-base.cc:116] --------------
I20260812 06:37:17.337369  8019 ts_itest-base.cc:117] 3 tablet servers
I20260812 06:37:17.337380  8019 ts_itest-base.cc:118] 3 replicas per TS
I20260812 06:37:17.337391  8019 ts_itest-base.cc:119] --------------
2026-08-12T06:37:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:17Z Disabled control of system clock
I20260812 06:37:17.347759  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:43595
--webserver_interface=127.7.212.254
--webserver_port=0
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.212.254:43595
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 06:37:17.453060 21491 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:17.453299 21491 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:17.453390 21491 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:17.456570 21491 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:17.456664 21491 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:17.456704 21491 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:17.456765 21491 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:17.460500 21491 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.7.212.254:43595
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.212.254:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.7.212.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:17.461520 21491 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:17.462634 21491 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:17.468169 21501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:17.468176 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:17.468181 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:17.468508 21491 server_base.cc:1061] running on GCE node
I20260812 06:37:17.469060 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:17.469937 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:17.471117 21491 hybrid_clock.cc:648] HybridClock initialized: now 1786516637471103 us; error 38 us; skew 500 ppm
I20260812 06:37:17.472735 21491 webserver.cc:533] Webserver started at http://127.7.212.254:44751/ using document root <none> and password file <none>
I20260812 06:37:17.473340 21491 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:17.473425 21491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:17.473642 21491 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:17.475093 21491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66b88020ebb242d3afb138c4a4abfa78"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.475584 21491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66b88020ebb242d3afb138c4a4abfa78"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.478701 21491 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.005s
I20260812 06:37:17.480729 21512 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.481684 21491 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:37:17.481817 21491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66b88020ebb242d3afb138c4a4abfa78"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.481910 21491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:17.510103 21491 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:17.510807 21491 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:17.511011 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:17.518729 21606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.254:43595 every 8 connection(s)
I20260812 06:37:17.518733 21491 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.254:43595
I20260812 06:37:17.519851 21491 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 06:37:17.522847 21607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:17.523847  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 21491
I20260812 06:37:17.523972  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 06:37:17.528523 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: Bootstrap starting.
I20260812 06:37:17.530866 21607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:17.531538 21607 log.cc:826] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:17.533223 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: No bootstrap required, opened a new log
I20260812 06:37:17.536046 21607 raft_consensus.cc:359] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } }
I20260812 06:37:17.536264 21607 raft_consensus.cc:385] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:17.536338 21607 raft_consensus.cc:740] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b88020ebb242d3afb138c4a4abfa78, State: Initialized, Role: FOLLOWER
I20260812 06:37:17.536751 21607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } }
I20260812 06:37:17.536919 21607 raft_consensus.cc:399] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 06:37:17.537007 21607 raft_consensus.cc:493] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 06:37:17.537122 21607 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:17.538122 21607 raft_consensus.cc:515] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } }
I20260812 06:37:17.538456 21607 leader_election.cc:304] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b88020ebb242d3afb138c4a4abfa78; no voters: 
I20260812 06:37:17.538744 21607 leader_election.cc:290] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 06:37:17.538887 21620 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:17.539110 21620 raft_consensus.cc:697] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [term 1 LEADER]: Becoming Leader. State: Replica: 66b88020ebb242d3afb138c4a4abfa78, State: Running, Role: LEADER
I20260812 06:37:17.539382 21620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } }
I20260812 06:37:17.539846 21607 sys_catalog.cc:565] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:17.540463 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b88020ebb242d3afb138c4a4abfa78. Latest consensus state: current_term: 1 leader_uuid: "66b88020ebb242d3afb138c4a4abfa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } } }
I20260812 06:37:17.540642 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:17.540889 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b88020ebb242d3afb138c4a4abfa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b88020ebb242d3afb138c4a4abfa78" member_type: VOTER last_known_addr { host: "127.7.212.254" port: 43595 } } }
I20260812 06:37:17.541010 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:17.541162 21633 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:17.543982 21633 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:17.549201 21633 catalog_manager.cc:1383] Generated new cluster ID: 98c3060dc434420c9a2064c1d79ad617
I20260812 06:37:17.549269 21633 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:17.562374 21633 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:17.563215 21633 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:17.569576 21633 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: Generated new TSK 0
I20260812 06:37:17.570210 21633 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 06:37:17.580662  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.193:0
--local_ip_for_outbound_sockets=127.7.212.193
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 06:37:17.689661 21658 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:17.689867 21658 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:17.689889 21658 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:17.689972 21658 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:17.693425 21658 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:17.693529 21658 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.193
I20260812 06:37:17.697316 21658 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.7.212.193
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--never_fsync=true
--heap_profile_path=/tmp/kudu.21658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.193

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:17.698380 21658 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:17.699522 21658 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:17.705974 21664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:17.705981 21663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:17.706233 21658 server_base.cc:1061] running on GCE node
W20260812 06:37:17.705981 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:17.706758 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:17.707327 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:17.708513 21658 hybrid_clock.cc:648] HybridClock initialized: now 1786516637708493 us; error 39 us; skew 500 ppm
I20260812 06:37:17.710603 21658 webserver.cc:533] Webserver started at http://127.7.212.193:43057/ using document root <none> and password file <none>
I20260812 06:37:17.711184 21658 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:17.711238 21658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:17.711437 21658 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:17.713099 21658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "72a20662eb944ee685b64fe704112774"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.713591 21658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "72a20662eb944ee685b64fe704112774"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.716945 21658 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:17.719340 21680 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.720428 21658 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:17.720590 21658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "72a20662eb944ee685b64fe704112774"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.720702 21658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:17.734447 21658 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:17.735152 21658 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:17.735376 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:17.735991 21658 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:17.737196 21658 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:17.737284 21658 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.737360 21658 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:17.737416 21658 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.748412 21658 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.193:32791
I20260812 06:37:17.748469 21885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.193:32791 every 8 connection(s)
I20260812 06:37:17.749541 21658 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 06:37:17.758100  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 21658
I20260812 06:37:17.758232  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 06:37:17.758865 21887 heartbeater.cc:344] Connected to a master server at 127.7.212.254:43595
I20260812 06:37:17.759128 21887 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:17.759735 21887 heartbeater.cc:507] Master 127.7.212.254:43595 requested a full tablet report, sending...
I20260812 06:37:17.760680  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:0
--local_ip_for_outbound_sockets=127.7.212.194
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 06:37:17.761417 21542 ts_manager.cc:194] Registered new tserver with Master: 72a20662eb944ee685b64fe704112774 (127.7.212.193:32791)
I20260812 06:37:17.763124 21542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.193:46951
W20260812 06:37:17.870471 21890 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:17.870781 21890 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:17.870855 21890 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:17.870973 21890 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:17.874431 21890 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:17.874593 21890 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:17.878363 21890 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--never_fsync=true
--heap_profile_path=/tmp/kudu.21890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:17.879521 21890 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:17.880671 21890 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:17.887105 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:17.887131 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:17.887131 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:17.887698 21890 server_base.cc:1061] running on GCE node
I20260812 06:37:17.888059 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:17.888595 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:17.889807 21890 hybrid_clock.cc:648] HybridClock initialized: now 1786516637889798 us; error 32 us; skew 500 ppm
I20260812 06:37:17.891574 21890 webserver.cc:533] Webserver started at http://127.7.212.194:43297/ using document root <none> and password file <none>
I20260812 06:37:17.892192 21890 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:17.892282 21890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:17.892511 21890 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:17.894164 21890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c371b9e2622842e69dc56bf3776e2979"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.894668 21890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c371b9e2622842e69dc56bf3776e2979"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.899919 21890 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20260812 06:37:17.902218 21917 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.903265 21890 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:17.903398 21890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c371b9e2622842e69dc56bf3776e2979"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:17.903501 21890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:17.921828 21890 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:17.922503 21890 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:17.922710 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:17.923269 21890 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:17.924274 21890 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:17.924351 21890 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.924423 21890 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:17.924468 21890 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:17.934377 21890 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:33083
I20260812 06:37:17.934408 22114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:33083 every 8 connection(s)
I20260812 06:37:17.935608 21890 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:17.939723  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 21890
I20260812 06:37:17.939877  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 06:37:17.942495  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:0
--local_ip_for_outbound_sockets=127.7.212.195
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 06:37:17.946056 22116 heartbeater.cc:344] Connected to a master server at 127.7.212.254:43595
I20260812 06:37:17.946336 22116 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:17.946977 22116 heartbeater.cc:507] Master 127.7.212.254:43595 requested a full tablet report, sending...
I20260812 06:37:17.948141 21542 ts_manager.cc:194] Registered new tserver with Master: c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083)
I20260812 06:37:17.948715 21542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:52899
W20260812 06:37:18.055989 22125 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:18.056201 22125 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:18.056247 22125 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:18.056356 22125 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:18.059767 22125 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:18.059865 22125 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:18.063601 22125 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=0
--tserver_master_addrs=127.7.212.254:43595
--never_fsync=true
--heap_profile_path=/tmp/kudu.22125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:18.064594 22125 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:18.065739 22125 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:18.071962 22135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:18.071967 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:18.071970 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:18.072414 22125 server_base.cc:1061] running on GCE node
I20260812 06:37:18.072824 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:18.073417 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:18.074638 22125 hybrid_clock.cc:648] HybridClock initialized: now 1786516638074615 us; error 42 us; skew 500 ppm
I20260812 06:37:18.076490 22125 webserver.cc:533] Webserver started at http://127.7.212.195:40505/ using document root <none> and password file <none>
I20260812 06:37:18.077188 22125 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:18.077282 22125 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:18.077494 22125 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:18.079103 22125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "31005dc958d342548e9475147f4a0c23"
format_stamp: "Formatted at 2026-08-12 06:37:18 on dist-test-slave-g170"
I20260812 06:37:18.079646 22125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "31005dc958d342548e9475147f4a0c23"
format_stamp: "Formatted at 2026-08-12 06:37:18 on dist-test-slave-g170"
I20260812 06:37:18.083739 22125 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260812 06:37:18.086074 22142 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:18.087102 22125 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:18.087246 22125 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "31005dc958d342548e9475147f4a0c23"
format_stamp: "Formatted at 2026-08-12 06:37:18 on dist-test-slave-g170"
I20260812 06:37:18.087364 22125 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:18.098918 22125 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:18.099507 22125 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:18.099699 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:18.100229 22125 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:18.101248 22125 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:18.101336 22125 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:18.101408 22125 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:18.101451 22125 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:18.110567 22125 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:37047
I20260812 06:37:18.110600 22346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:37047 every 8 connection(s)
I20260812 06:37:18.113032 22125 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:18.121167  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 22125
I20260812 06:37:18.121317  8019 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 06:37:18.122056 22349 heartbeater.cc:344] Connected to a master server at 127.7.212.254:43595
I20260812 06:37:18.122344 22349 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:18.122926 22349 heartbeater.cc:507] Master 127.7.212.254:43595 requested a full tablet report, sending...
I20260812 06:37:18.124039 21542 ts_manager.cc:194] Registered new tserver with Master: 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047)
I20260812 06:37:18.124583 21542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:37013
I20260812 06:37:18.135648  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:18.145207 21542 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52902:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 06:37:18.147625 21542 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 06:37:18.163892 22224 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 (DEFAULT_TABLE table=TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:18.164914 22224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa61d65e1684c588ba27ad5c3a626d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:18.164521 22002 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 (DEFAULT_TABLE table=TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:18.164521 21769 tablet_service.cc:1511] Processing CreateTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 (DEFAULT_TABLE table=TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 06:37:18.165503 21769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa61d65e1684c588ba27ad5c3a626d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:18.165503 22002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa61d65e1684c588ba27ad5c3a626d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:18.169826 22372 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Bootstrap starting.
I20260812 06:37:18.170989 22373 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap starting.
I20260812 06:37:18.172968 22381 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Bootstrap starting.
I20260812 06:37:18.172999 22373 tablet_bootstrap.cc:654] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:18.172999 22372 tablet_bootstrap.cc:654] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:18.173928 22373 log.cc:826] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:18.173928 22372 log.cc:826] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:18.174899 22381 tablet_bootstrap.cc:654] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:18.175542 22381 log.cc:826] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:18.175679 22373 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: No bootstrap required, opened a new log
I20260812 06:37:18.175679 22372 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: No bootstrap required, opened a new log
I20260812 06:37:18.175868 22373 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:18.175868 22372 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 06:37:18.177109 22381 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: No bootstrap required, opened a new log
I20260812 06:37:18.177315 22381 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20260812 06:37:18.178997 22373 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.178996 22372 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.179222 22372 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31005dc958d342548e9475147f4a0c23, State: Initialized, Role: FOLLOWER
I20260812 06:37:18.179222 22373 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c371b9e2622842e69dc56bf3776e2979, State: Initialized, Role: FOLLOWER
I20260812 06:37:18.179548 22373 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.179548 22372 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.179845 22381 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.180054 22381 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a20662eb944ee685b64fe704112774, State: Initialized, Role: FOLLOWER
I20260812 06:37:18.180392 22373 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:37:18.180392 22372 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:37:18.180483 22349 heartbeater.cc:499] Master 127.7.212.254:43595 was elected leader, sending a full tablet report...
I20260812 06:37:18.180698 22116 heartbeater.cc:499] Master 127.7.212.254:43595 was elected leader, sending a full tablet report...
I20260812 06:37:18.180354 22381 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.182077 22381 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:18.182395 21887 heartbeater.cc:499] Master 127.7.212.254:43595 was elected leader, sending a full tablet report...
W20260812 06:37:18.186527 22117 tablet.cc:2406] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:18.193910  8019 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 06:37:18.195319  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 72a20662eb944ee685b64fe704112774 to finish bootstrapping
I20260812 06:37:18.200946  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c371b9e2622842e69dc56bf3776e2979 to finish bootstrapping
I20260812 06:37:18.206259  8019 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 31005dc958d342548e9475147f4a0c23 to finish bootstrapping
I20260812 06:37:18.214792 21809 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4"
dest_uuid: "72a20662eb944ee685b64fe704112774"
 from {username='slave'} at 127.0.0.1:58454
I20260812 06:37:18.215018 21809 raft_consensus.cc:493] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 06:37:18.215135 21809 raft_consensus.cc:3060] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:18.216054 21809 raft_consensus.cc:515] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.217163 21809 leader_election.cc:290] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [CANDIDATE]: Term 1 election: Requested vote from peers c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083), 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047)
I20260812 06:37:18.221665 22037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4" candidate_uuid: "72a20662eb944ee685b64fe704112774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c371b9e2622842e69dc56bf3776e2979"
I20260812 06:37:18.221908 22037 raft_consensus.cc:3060] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:18.222081 22261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4" candidate_uuid: "72a20662eb944ee685b64fe704112774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31005dc958d342548e9475147f4a0c23"
I20260812 06:37:18.222286 22261 raft_consensus.cc:3060] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:18.223209 22037 raft_consensus.cc:2468] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a20662eb944ee685b64fe704112774 in term 1.
I20260812 06:37:18.223663 22261 raft_consensus.cc:2468] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a20662eb944ee685b64fe704112774 in term 1.
I20260812 06:37:18.223721 21682 leader_election.cc:304] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72a20662eb944ee685b64fe704112774, c371b9e2622842e69dc56bf3776e2979; no voters: 
I20260812 06:37:18.223984 22391 raft_consensus.cc:2804] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:18.224171 22391 raft_consensus.cc:697] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: Becoming Leader. State: Replica: 72a20662eb944ee685b64fe704112774, State: Running, Role: LEADER
I20260812 06:37:18.224512 22391 consensus_queue.cc:237] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.227953 21538 catalog_manager.cc:5719] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72a20662eb944ee685b64fe704112774 (127.7.212.193). New cstate: current_term: 1 leader_uuid: "72a20662eb944ee685b64fe704112774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20260812 06:37:18.250802 21888 tablet.cc:2406] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:37:18.364144 22350 tablet.cc:2406] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:18.671569 22391 consensus_queue.cc:1048] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:18.677305 22407 consensus_queue.cc:1048] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:18.688723 22104 generic_service.cc:175] {username='slave'} at 127.0.0.1:54192 changed flags via RPC: fault_crash_before_append_commit from '0' to '1.0'
I20260812 06:37:18.689913 22337 generic_service.cc:175] {username='slave'} at 127.0.0.1:40400 changed flags via RPC: fault_crash_before_append_commit from '0' to '1.0'
I20260812 06:37:18.691424 21809 consensus_queue.cc:237] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:18.692695 22261 raft_consensus.cc:1275] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Refusing update from remote peer 72a20662eb944ee685b64fe704112774: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:18.693096 22399 consensus_queue.cc:1048] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260812 06:37:18.694700 22407 raft_consensus.cc:2955] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c371b9e2622842e69dc56bf3776e2979 (127.7.212.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } }
I20260812 06:37:18.695273 22261 raft_consensus.cc:2955] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c371b9e2622842e69dc56bf3776e2979 (127.7.212.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } }
W20260812 06:37:18.696466 22261 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 06:37:18.696506 22261 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 06:37:18.696523 22261 fault_injection.cc:59] injecting fault for kudu (pid 22125): FLAGS_fault_crash_before_append_commit (process will exit)
I20260812 06:37:18.705317 21538 catalog_manager.cc:5719] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 reported cstate change: config changed from index -1 to 2, VOTER c371b9e2622842e69dc56bf3776e2979 (127.7.212.194) evicted. New cstate: current_term: 1 leader_uuid: "72a20662eb944ee685b64fe704112774" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } health_report { overall_health: HEALTHY } } }
I20260812 06:37:18.708608 22002 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 with delete_type TABLET_DATA_TOMBSTONED (TS c371b9e2622842e69dc56bf3776e2979 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:54180
I20260812 06:37:18.709300 22414 tablet_replica.cc:333] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: stopping tablet replica
I20260812 06:37:18.709520 21809 consensus_queue.cc:237] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: NON_VOTER last_known_addr { host: "127.7.212.194" port: 33083 } attrs { promote: true } }
I20260812 06:37:18.709618 22414 raft_consensus.cc:2243] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 06:37:18.709900 22414 raft_consensus.cc:2272] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 06:37:18.711273 22414 ts_tablet_manager.cc:1916] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260812 06:37:18.711743 21682 consensus_peers.cc:597] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 -> Peer c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083): Couldn't send request to peer c371b9e2622842e69dc56bf3776e2979. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260812 06:37:18.712291 21682 connection.cc:573] client connection to 127.7.212.195:37047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:37:18.712421 21682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 06:37:18.713572 21682 consensus_peers.cc:597] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 -> Peer 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047): Couldn't send request to peer 31005dc958d342548e9475147f4a0c23. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.195:37047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 06:37:18.713729 22414 ts_tablet_manager.cc:1929] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260812 06:37:18.713820 22414 log.cc:1199] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting WAL directory at /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:18.714412 21513 catalog_manager.cc:5050] TS c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083): tablet 8aa61d65e1684c588ba27ad5c3a626d4 (table TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]) successfully deleted
I20260812 06:37:19.638152 22416 ts_tablet_manager.cc:933] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Initiating tablet copy from peer 72a20662eb944ee685b64fe704112774 (127.7.212.193:32791)
I20260812 06:37:19.638790 22416 tablet_copy_client.cc:323] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet copy: Beginning tablet copy session from remote peer at address 127.7.212.193:32791
I20260812 06:37:19.643450 21847 tablet_copy_service.cc:140] P 72a20662eb944ee685b64fe704112774: Received BeginTabletCopySession request for tablet 8aa61d65e1684c588ba27ad5c3a626d4 from peer c371b9e2622842e69dc56bf3776e2979 ({username='slave'} at 127.7.212.194:59939)
I20260812 06:37:19.643644 21847 tablet_copy_service.cc:161] P 72a20662eb944ee685b64fe704112774: Beginning new tablet copy session on tablet 8aa61d65e1684c588ba27ad5c3a626d4 from peer c371b9e2622842e69dc56bf3776e2979 at {username='slave'} at 127.7.212.194:59939: session id = c371b9e2622842e69dc56bf3776e2979-8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:19.644891 21847 tablet_copy_source_session.cc:215] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 06:37:19.646301 22416 ts_tablet_manager.cc:1916] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260812 06:37:19.648109 22416 ts_tablet_manager.cc:1929] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260812 06:37:19.648303 22416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa61d65e1684c588ba27ad5c3a626d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:19.649464 22416 tablet_copy_client.cc:806] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet copy: Starting download of 0 data blocks...
I20260812 06:37:19.649684 22416 tablet_copy_client.cc:670] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet copy: Starting download of 1 WAL segments...
I20260812 06:37:19.651711 22416 tablet_copy_client.cc:538] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 06:37:19.653298 22416 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap starting.
I20260812 06:37:19.657678 22416 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 06:37:19.658016 22416 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap complete.
I20260812 06:37:19.658192 22416 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 06:37:19.658748 22416 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:19.659009 22416 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c371b9e2622842e69dc56bf3776e2979, State: Initialized, Role: LEARNER
I20260812 06:37:19.659150 22416 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: NON_VOTER last_known_addr { host: "127.7.212.194" port: 33083 } attrs { promote: true } }
I20260812 06:37:19.659575 22416 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent starting tablet: real 0.001s	user 0.003s	sys 0.000s
I20260812 06:37:19.660354 21847 tablet_copy_service.cc:342] P 72a20662eb944ee685b64fe704112774: Request end of tablet copy session c371b9e2622842e69dc56bf3776e2979-8aa61d65e1684c588ba27ad5c3a626d4 received from {username='slave'} at 127.7.212.194:59939
I20260812 06:37:19.660488 21847 tablet_copy_service.cc:434] P 72a20662eb944ee685b64fe704112774: ending tablet copy session c371b9e2622842e69dc56bf3776e2979-8aa61d65e1684c588ba27ad5c3a626d4 on tablet 8aa61d65e1684c588ba27ad5c3a626d4 with peer c371b9e2622842e69dc56bf3776e2979
I20260812 06:37:20.244552 22037 raft_consensus.cc:1217] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20260812 06:37:20.849848 22430 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260812 06:37:21.272543 21682 consensus_peers.cc:597] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 -> Peer 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047): Couldn't send request to peer 31005dc958d342548e9475147f4a0c23. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.195:37047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 06:37:21.334794 22434 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 06:37:21.936381 22434 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 06:37:22.376482 22440 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 06:37:22.991093 22441 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260812 06:37:23.533744 22446 raft_consensus.cc:1027] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: attempt to promote peer c371b9e2622842e69dc56bf3776e2979: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260812 06:37:23.660044 21682 consensus_peers.cc:597] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 -> Peer 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047): Couldn't send request to peer 31005dc958d342548e9475147f4a0c23. Status: Network error: Client connection negotiation failed: client connection to 127.7.212.195:37047: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260812 06:37:23.812058 21540 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52902:
table { table_name: "TestTable" } modify_external_catalogs: true
I20260812 06:37:23.812299 21540 catalog_manager.cc:2781] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52902:
table { table_name: "TestTable" } modify_external_catalogs: true
I20260812 06:37:23.815954 21540 catalog_manager.cc:6006] T 00000000000000000000000000000000 P 66b88020ebb242d3afb138c4a4abfa78: Sending DeleteTablet for 2 replicas of tablet 8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:23.816874  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 21890
I20260812 06:37:23.816843 21769 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 06:37:23 UTC) from {username='slave'} at 127.0.0.1:58438
W20260812 06:37:23.829043 21513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.212.195:37047: connect: Connection refused (error 111)
W20260812 06:37:23.832158 21513 catalog_manager.cc:4777] TS 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8aa61d65e1684c588ba27ad5c3a626d4: Network error: Client connection negotiation failed: client connection to 127.7.212.195:37047: connect: Connection refused (error 111)
I20260812 06:37:23.833621 22462 tablet_replica.cc:333] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: stopping tablet replica
I20260812 06:37:23.834009 22462 raft_consensus.cc:2243] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: Raft consensus shutting down.
I20260812 06:37:23.834359 22462 pending_rounds.cc:70] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Trying to abort 1 pending ops.
I20260812 06:37:23.834488 22462 pending_rounds.cc:77] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7317572152153378816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: NON_VOTER last_known_addr { host: "127.7.212.194" port: 33083 } attrs { promote: true } } } }
I20260812 06:37:23.834645 22462 raft_consensus.cc:2917] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260812 06:37:23.834750 22462 raft_consensus.cc:2889] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260812 06:37:23.835031 22462 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:58438 (request call id 1) took 5125 ms. Trace:
I20260812 06:37:23.835206 22462 rpcz_store.cc:276] 0812 06:37:18.709081 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 06:37:18.709159 (+    78us) service_pool.cc:224] Handling call
0812 06:37:18.710070 (+   911us) log.cc:844] Serialized 374 byte log entry
0812 06:37:23.835014 (+5124944us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":346,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":213}
I20260812 06:37:23.835851 22462 raft_consensus.cc:2272] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 06:37:23.836968 22462 ts_tablet_manager.cc:1916] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 06:37:23.839278 22462 ts_tablet_manager.cc:1929] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260812 06:37:23.839414 22462 log.cc:1199] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Deleting WAL directory at /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-0/wal/wals/8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:23.839761 22462 ts_tablet_manager.cc:1950] T 8aa61d65e1684c588ba27ad5c3a626d4 P 72a20662eb944ee685b64fe704112774: Deleting consensus metadata
I20260812 06:37:23.841272 21518 catalog_manager.cc:5050] TS 72a20662eb944ee685b64fe704112774 (127.7.212.193:32791): tablet 8aa61d65e1684c588ba27ad5c3a626d4 (table TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]) successfully deleted
I20260812 06:37:23.841718  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.194:33083
--local_ip_for_outbound_sockets=127.7.212.194
--tserver_master_addrs=127.7.212.254:43595
--webserver_port=43297
--webserver_interface=127.7.212.194
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 06:37:23.953059 22466 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:23.953358 22466 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:23.953435 22466 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:23.953522 22466 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:23.956960 22466 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:23.957140 22466 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.194
I20260812 06:37:23.961100 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.194:33083
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.7.212.194
--webserver_port=43297
--tserver_master_addrs=127.7.212.254:43595
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.194

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:23.962244 22466 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:23.963443 22466 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:23.970291 22476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:23.970356 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:23.970569 22466 server_base.cc:1061] running on GCE node
W20260812 06:37:23.970453 22483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:23.971086 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:23.971642 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:23.972863 22466 hybrid_clock.cc:648] HybridClock initialized: now 1786516643972851 us; error 24 us; skew 500 ppm
I20260812 06:37:23.974599 22466 webserver.cc:533] Webserver started at http://127.7.212.194:43297/ using document root <none> and password file <none>
I20260812 06:37:23.975171 22466 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:23.975274 22466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:23.978613 22466 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.000s	sys 0.003s
I20260812 06:37:23.980494 22495 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:23.981585 22466 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 06:37:23.981717 22466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c371b9e2622842e69dc56bf3776e2979"
format_stamp: "Formatted at 2026-08-12 06:37:17 on dist-test-slave-g170"
I20260812 06:37:23.982126 22466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:24.001626 22466 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:24.002374 22466 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:24.002600 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:24.003232 22466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:24.004544 22514 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:24.008109 22466 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:24.008198 22466 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:37:24.008304 22466 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:24.010075 22466 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:24.010171 22466 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.002s
I20260812 06:37:24.010610 22514 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap starting.
I20260812 06:37:24.020992 22466 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.194:33083
I20260812 06:37:24.022440 22466 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 06:37:24.024731 22710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.194:33083 every 8 connection(s)
I20260812 06:37:24.029093  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 22466
I20260812 06:37:24.029733  8019 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
/tmp/dist-test-task0q5Kdv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.212.195:37047
--local_ip_for_outbound_sockets=127.7.212.195
--tserver_master_addrs=127.7.212.254:43595
--webserver_port=40505
--webserver_interface=127.7.212.195
--builtin_ntp_servers=127.7.212.212:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 06:37:24.031220 22514 log.cc:826] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:24.035037 22514 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 06:37:24.035557 22514 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Bootstrap complete.
I20260812 06:37:24.036895 22514 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent bootstrapping tablet: real 0.027s	user 0.025s	sys 0.000s
I20260812 06:37:24.040165 22514 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:24.041013 22514 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c371b9e2622842e69dc56bf3776e2979, State: Initialized, Role: LEARNER
I20260812 06:37:24.041492 22514 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: NON_VOTER last_known_addr { host: "127.7.212.194" port: 33083 } attrs { promote: true } }
I20260812 06:37:24.041754 22712 heartbeater.cc:344] Connected to a master server at 127.7.212.254:43595
I20260812 06:37:24.042079 22712 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:24.042502 22514 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 06:37:24.042752 22712 heartbeater.cc:507] Master 127.7.212.254:43595 requested a full tablet report, sending...
I20260812 06:37:24.045086 21540 ts_manager.cc:194] Re-registered known tserver with Master: c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083)
I20260812 06:37:24.046459 21540 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.194:56457
I20260812 06:37:24.048830 22712 heartbeater.cc:499] Master 127.7.212.254:43595 was elected leader, sending a full tablet report...
I20260812 06:37:24.052830 22593 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 06:37:23 UTC) from {username='slave'} at 127.0.0.1:54208
I20260812 06:37:24.053691 22723 tablet_replica.cc:333] stopping tablet replica
I20260812 06:37:24.053956 22723 raft_consensus.cc:2243] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 LEARNER]: Raft consensus shutting down.
I20260812 06:37:24.054181 22723 pending_rounds.cc:70] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Trying to abort 1 pending ops.
I20260812 06:37:24.054261 22723 pending_rounds.cc:77] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7317572152153378816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: NON_VOTER last_known_addr { host: "127.7.212.194" port: 33083 } attrs { promote: true } } } }
I20260812 06:37:24.054440 22723 raft_consensus.cc:2917] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 LEARNER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260812 06:37:24.054534 22723 raft_consensus.cc:2889] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260812 06:37:24.054595 22723 raft_consensus.cc:2272] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260812 06:37:24.055596 22723 ts_tablet_manager.cc:1916] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 06:37:24.059266 22723 ts_tablet_manager.cc:1929] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260812 06:37:24.059360 22723 log.cc:1199] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting WAL directory at /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-1/wal/wals/8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:24.059630 22723 ts_tablet_manager.cc:1950] T 8aa61d65e1684c588ba27ad5c3a626d4 P c371b9e2622842e69dc56bf3776e2979: Deleting consensus metadata
I20260812 06:37:24.061297 21513 catalog_manager.cc:5050] TS c371b9e2622842e69dc56bf3776e2979 (127.7.212.194:33083): tablet 8aa61d65e1684c588ba27ad5c3a626d4 (table TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]) successfully deleted
W20260812 06:37:24.144029 22718 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 06:37:24.144249 22718 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:24.144271 22718 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:24.144369 22718 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:24.147707 22718 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:24.147850 22718 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.212.195
I20260812 06:37:24.151520 22718 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.212.212:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.212.195:37047
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.7.212.195
--webserver_port=40505
--tserver_master_addrs=127.7.212.254:43595
--never_fsync=true
--heap_profile_path=/tmp/kudu.22718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.212.195

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:24.152541 22718 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:24.153697 22718 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:24.160048 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:24.160058 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:24.160249 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:24.160290 22718 server_base.cc:1061] running on GCE node
I20260812 06:37:24.160749 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:24.161324 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:24.162562 22718 hybrid_clock.cc:648] HybridClock initialized: now 1786516644162529 us; error 45 us; skew 500 ppm
I20260812 06:37:24.164538 22718 webserver.cc:533] Webserver started at http://127.7.212.195:40505/ using document root <none> and password file <none>
I20260812 06:37:24.165230 22718 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:24.165341 22718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:24.169204 22718 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:24.171258 22751 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:24.172421 22718 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:24.172598 22718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "31005dc958d342548e9475147f4a0c23"
format_stamp: "Formatted at 2026-08-12 06:37:18 on dist-test-slave-g170"
I20260812 06:37:24.173180 22718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:24.189824 22718 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:24.190579 22718 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:24.190821 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:24.191479 22718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:24.192911 22760 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 06:37:24.196224 22718 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 06:37:24.196326 22718 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
I20260812 06:37:24.196414 22718 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 06:37:24.198136 22718 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 06:37:24.198210 22718 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:24.198523 22760 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Bootstrap starting.
I20260812 06:37:24.210851 22718 rpc_server.cc:307] RPC server started. Bound to: 127.7.212.195:37047
I20260812 06:37:24.210935 22956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.212.195:37047 every 8 connection(s)
I20260812 06:37:24.211946 22718 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 06:37:24.213529 22760 log.cc:826] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:24.216298 22760 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 06:37:24.216689 22760 tablet_bootstrap.cc:492] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Bootstrap complete.
I20260812 06:37:24.218147 22760 ts_tablet_manager.cc:1403] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.008s
I20260812 06:37:24.218562  8019 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu as pid 22718
I20260812 06:37:24.221268 22957 heartbeater.cc:344] Connected to a master server at 127.7.212.254:43595
I20260812 06:37:24.221519 22957 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:24.221602 22760 raft_consensus.cc:359] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:24.222056 22957 heartbeater.cc:507] Master 127.7.212.254:43595 requested a full tablet report, sending...
I20260812 06:37:24.222083 22760 raft_consensus.cc:886] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Ignoring setting pending config change with OpId 1.2 because the committed config has OpId index 2. The config change we are ignoring is: Old config: { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } }. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } }
I20260812 06:37:24.222330 22760 raft_consensus.cc:740] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31005dc958d342548e9475147f4a0c23, State: Initialized, Role: FOLLOWER
I20260812 06:37:24.222644 22760 consensus_queue.cc:260] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } }
I20260812 06:37:24.223481 22760 ts_tablet_manager.cc:1434] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 06:37:24.224466 21540 ts_manager.cc:194] Re-registered known tserver with Master: 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047)
I20260812 06:37:24.225378 21540 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.212.195:60437
I20260812 06:37:24.227594 22957 heartbeater.cc:499] Master 127.7.212.254:43595 was elected leader, sending a full tablet report...
I20260812 06:37:24.498528 22837 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aa61d65e1684c588ba27ad5c3a626d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 06:37:23 UTC) from {username='slave'} at 127.0.0.1:40510
I20260812 06:37:24.499454 22970 tablet_replica.cc:333] stopping tablet replica
I20260812 06:37:24.499740 22970 raft_consensus.cc:2243] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 06:37:24.499959 22970 pending_rounds.cc:70] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Trying to abort 1 pending ops.
I20260812 06:37:24.500036 22970 pending_rounds.cc:77] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7317572152078639104 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8aa61d65e1684c588ba27ad5c3a626d4" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "c371b9e2622842e69dc56bf3776e2979" member_type: VOTER last_known_addr { host: "127.7.212.194" port: 33083 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } } new_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "72a20662eb944ee685b64fe704112774" member_type: VOTER last_known_addr { host: "127.7.212.193" port: 32791 } } peers { permanent_uuid: "31005dc958d342548e9475147f4a0c23" member_type: VOTER last_known_addr { host: "127.7.212.195" port: 37047 } } } }
I20260812 06:37:24.500178 22970 raft_consensus.cc:2925] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Skipping abort of non-pending config change with OpId 1.2: Aborted: Op aborted
I20260812 06:37:24.500218 22970 raft_consensus.cc:2889] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260812 06:37:24.500262 22970 raft_consensus.cc:2272] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 06:37:24.501196 22970 ts_tablet_manager.cc:1916] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 06:37:24.504076 22970 ts_tablet_manager.cc:1929] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 06:37:24.504161 22970 log.cc:1199] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Deleting WAL directory at /tmp/dist-test-task0q5Kdv/test-tmp/raft_consensus-itest.5.RaftConsensusParamReplicationModesITest.Test_KUDU_1735_1.1786516611938192-8019-0/raft_consensus-itest-cluster/ts-2/wal/wals/8aa61d65e1684c588ba27ad5c3a626d4
I20260812 06:37:24.504465 22970 ts_tablet_manager.cc:1950] T 8aa61d65e1684c588ba27ad5c3a626d4 P 31005dc958d342548e9475147f4a0c23: Deleting consensus metadata
I20260812 06:37:24.505698 21513 catalog_manager.cc:5050] TS 31005dc958d342548e9475147f4a0c23 (127.7.212.195:37047): tablet 8aa61d65e1684c588ba27ad5c3a626d4 (table TestTable [id=7e6bc0c4e2ad49d29a61b3fa0631f52f]) successfully deleted
I20260812 06:37:24.515882  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 21658
I20260812 06:37:24.530612  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 22466
I20260812 06:37:24.544523  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 22718
I20260812 06:37:24.558748  8019 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0q5Kdv/build/debug/bin/kudu with pid 21491
2026-08-12T06:37:24Z chronyd exiting
[       OK ] RaftConsensusParamReplicationModesITest.Test_KUDU_1735/1 (7254 ms)
[----------] 1 test from RaftConsensusParamReplicationModesITest (7254 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 2 test suites ran. (32640 ms total)
[  PASSED  ] 7 tests.
