Note: This is test shard 2 of 6.
[==========] Running 8 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from RaftConsensusITest
[ RUN      ] RaftConsensusITest.TestInsertAndMutateThroughConsensus
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 08:00:25.855000 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:00:25.855166 23362 ts_itest-base.cc:116] --------------
I20260812 08:00:25.855180 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:00:25.855193 23362 ts_itest-base.cc:118] 3 replicas per TS
I20260812 08:00:25.855204 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:00:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:00:25Z Disabled control of system clock
I20260812 08:00:25.867681 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:35703
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:35703 with env {}
W20260812 08:00:25.985898 23377 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:25.986151 23377 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:25.986243 23377 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:25.990393 23377 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:00:25.990480 23377 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:25.990504 23377 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:00:25.990525 23377 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:00:25.995414 23377 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:35703
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:35703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:25.996874 23377 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:25.998335 23377 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:26.005059 23382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.005334 23377 server_base.cc:1061] running on GCE node
W20260812 08:00:26.005409 23383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.005059 23385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.006170 23377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:26.007345 23377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:26.008522 23377 hybrid_clock.cc:648] HybridClock initialized: now 1786521626008491 us; error 106 us; skew 500 ppm
I20260812 08:00:26.010803 23377 webserver.cc:533] Webserver started at http://127.22.208.190:45817/ using document root <none> and password file <none>
I20260812 08:00:26.011498 23377 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:26.011564 23377 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:26.011806 23377 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:26.013623 23377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7ea33df98d1f4b19814a4389ae0fe3d9"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.014389 23377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7ea33df98d1f4b19814a4389ae0fe3d9"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.018728 23377 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.003s
I20260812 08:00:26.021441 23392 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.023759 23377 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.003s
I20260812 08:00:26.023970 23377 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7ea33df98d1f4b19814a4389ae0fe3d9"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.024135 23377 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:26.067631 23377 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:26.068634 23377 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:26.068881 23377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:26.077570 23377 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:35703
I20260812 08:00:26.077698 23458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:35703 every 8 connection(s)
I20260812 08:00:26.078825 23377 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:00:26.082203 23459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:26.086800 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 23377
I20260812 08:00:26.086977 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:00:26.089051 23459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: Bootstrap starting.
I20260812 08:00:26.092387 23459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:26.093432 23459 log.cc:826] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:26.095685 23459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: No bootstrap required, opened a new log
I20260812 08:00:26.099021 23459 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } }
I20260812 08:00:26.099311 23459 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:26.099395 23459 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea33df98d1f4b19814a4389ae0fe3d9, State: Initialized, Role: FOLLOWER
I20260812 08:00:26.099997 23459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } }
I20260812 08:00:26.100199 23459 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:00:26.100272 23459 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:00:26.100409 23459 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:26.101413 23459 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } }
I20260812 08:00:26.101783 23459 leader_election.cc:304] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ea33df98d1f4b19814a4389ae0fe3d9; no voters: 
I20260812 08:00:26.102077 23459 leader_election.cc:290] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:00:26.102252 23465 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:26.102514 23465 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7ea33df98d1f4b19814a4389ae0fe3d9, State: Running, Role: LEADER
I20260812 08:00:26.102882 23465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } }
I20260812 08:00:26.103338 23459 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:00:26.104784 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ea33df98d1f4b19814a4389ae0fe3d9. Latest consensus state: current_term: 1 leader_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } } }
I20260812 08:00:26.104931 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:26.105897 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea33df98d1f4b19814a4389ae0fe3d9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35703 } } }
I20260812 08:00:26.106011 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9 [sys.catalog]: This master's current role is: LEADER
W20260812 08:00:26.108980 23482 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:00:26.109094 23482 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:00:26.109182 23474 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:00:26.110121 23474 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:00:26.116431 23474 catalog_manager.cc:1383] Generated new cluster ID: bfb48f6e1b1344f587d6714d79072c46
I20260812 08:00:26.116554 23474 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:00:26.137667 23474 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:00:26.138702 23474 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:00:26.154280 23474 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 7ea33df98d1f4b19814a4389ae0fe3d9: Generated new TSK 0
I20260812 08:00:26.155208 23474 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:00:26.166978 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--builtin_ntp_servers=127.22.208.148:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260812 08:00:26.285123 23486 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:26.285396 23486 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:26.285441 23486 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:26.289063 23486 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:26.289197 23486 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:00:26.293413 23486 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--never_fsync=true
--heap_profile_path=/tmp/kudu.23486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:26.294570 23486 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:26.295732 23486 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:26.303092 23494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.303092 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.303189 23486 server_base.cc:1061] running on GCE node
W20260812 08:00:26.303120 23493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.303884 23486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:26.304435 23486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:26.305603 23486 hybrid_clock.cc:648] HybridClock initialized: now 1786521626305580 us; error 41 us; skew 500 ppm
I20260812 08:00:26.307664 23486 webserver.cc:533] Webserver started at http://127.22.208.129:41009/ using document root <none> and password file <none>
I20260812 08:00:26.308323 23486 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:26.308385 23486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:26.308593 23486 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:26.310364 23486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0db50616c97e4a9c9c298ed2df692dc1"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.310916 23486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0db50616c97e4a9c9c298ed2df692dc1"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.314437 23486 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:26.316797 23504 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.318033 23486 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:26.318185 23486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0db50616c97e4a9c9c298ed2df692dc1"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.318325 23486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:26.354503 23486 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:26.355356 23486 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:26.355615 23486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:26.356314 23486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:26.357452 23486 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:26.357528 23486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.357635 23486 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:26.357870 23486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.368680 23486 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:40449
I20260812 08:00:26.368708 23635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:40449 every 8 connection(s)
I20260812 08:00:26.369721 23486 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:00:26.374996 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 23486
I20260812 08:00:26.375097 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:00:26.377740 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--builtin_ntp_servers=127.22.208.148:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 08:00:26.381772 23636 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35703
I20260812 08:00:26.382045 23636 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:26.382685 23636 heartbeater.cc:507] Master 127.22.208.190:35703 requested a full tablet report, sending...
I20260812 08:00:26.384290 23414 ts_manager.cc:194] Registered new tserver with Master: 0db50616c97e4a9c9c298ed2df692dc1 (127.22.208.129:40449)
I20260812 08:00:26.385454 23414 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:53877
W20260812 08:00:26.494434 23639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:26.494786 23639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:26.494851 23639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:26.498299 23639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:26.498431 23639 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:00:26.502454 23639 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--never_fsync=true
--heap_profile_path=/tmp/kudu.23639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:26.503568 23639 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:26.504724 23639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:26.512290 23644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.512276 23647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.512292 23645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.512946 23639 server_base.cc:1061] running on GCE node
I20260812 08:00:26.513468 23639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:26.514231 23639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:26.515501 23639 hybrid_clock.cc:648] HybridClock initialized: now 1786521626515280 us; error 245 us; skew 500 ppm
I20260812 08:00:26.517656 23639 webserver.cc:533] Webserver started at http://127.22.208.130:37577/ using document root <none> and password file <none>
I20260812 08:00:26.518285 23639 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:26.518370 23639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:26.518661 23639 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:26.520341 23639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a86c753e99aa42f5a61e8cff2e2581d6"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.520854 23639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a86c753e99aa42f5a61e8cff2e2581d6"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.524698 23639 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.002s
I20260812 08:00:26.527230 23653 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.528357 23639 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:00:26.528527 23639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a86c753e99aa42f5a61e8cff2e2581d6"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.528662 23639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:26.563158 23639 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:26.563915 23639 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:26.564126 23639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:26.564743 23639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:26.565817 23639 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:26.565888 23639 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.565960 23639 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:26.566007 23639 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.576586 23639 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:39217
I20260812 08:00:26.576629 23787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:39217 every 8 connection(s)
I20260812 08:00:26.577610 23639 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:00:26.586777 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 23639
I20260812 08:00:26.586920 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:00:26.588063 23788 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35703
I20260812 08:00:26.588352 23788 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:26.588984 23788 heartbeater.cc:507] Master 127.22.208.190:35703 requested a full tablet report, sending...
I20260812 08:00:26.589601 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--builtin_ntp_servers=127.22.208.148:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 08:00:26.590324 23414 ts_manager.cc:194] Registered new tserver with Master: a86c753e99aa42f5a61e8cff2e2581d6 (127.22.208.130:39217)
I20260812 08:00:26.591033 23414 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:58965
W20260812 08:00:26.709744 23791 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:26.710038 23791 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:26.710114 23791 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:26.713758 23791 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:26.713965 23791 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:00:26.718104 23791 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35703
--never_fsync=true
--heap_profile_path=/tmp/kudu.23791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:26.719389 23791 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:26.720561 23791 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:26.727703 23796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.727792 23799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:26.727703 23797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:26.728061 23791 server_base.cc:1061] running on GCE node
I20260812 08:00:26.728477 23791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:26.729043 23791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:26.730283 23791 hybrid_clock.cc:648] HybridClock initialized: now 1786521626730260 us; error 32 us; skew 500 ppm
I20260812 08:00:26.732334 23791 webserver.cc:533] Webserver started at http://127.22.208.131:36727/ using document root <none> and password file <none>
I20260812 08:00:26.732903 23791 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:26.732963 23791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:26.733142 23791 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:26.734980 23791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bf1a83fcf0fb4125a1765ee32d812593"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.735488 23791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bf1a83fcf0fb4125a1765ee32d812593"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.739495 23791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:00:26.742194 23805 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.743579 23791 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:26.743831 23791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bf1a83fcf0fb4125a1765ee32d812593"
format_stamp: "Formatted at 2026-08-12 08:00:26 on dist-test-slave-dhph"
I20260812 08:00:26.743964 23791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:26.756937 23791 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:26.757656 23791 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:26.757879 23791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:26.758497 23791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:26.759658 23791 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:26.759748 23791 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.759794 23791 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:26.759810 23791 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:26.770800 23791 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:39731
I20260812 08:00:26.770892 23931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:39731 every 8 connection(s)
I20260812 08:00:26.772274 23791 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:00:26.780288 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 23791
I20260812 08:00:26.780411 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:00:26.782950 23932 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35703
I20260812 08:00:26.783272 23932 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:26.783902 23932 heartbeater.cc:507] Master 127.22.208.190:35703 requested a full tablet report, sending...
I20260812 08:00:26.785081 23414 ts_manager.cc:194] Registered new tserver with Master: bf1a83fcf0fb4125a1765ee32d812593 (127.22.208.131:39731)
I20260812 08:00:26.785740 23414 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:33107
I20260812 08:00:26.796243 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:26.811142 23414 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57906:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 08:00:26.813553 23414 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:00:26.832276 23561 tablet_service.cc:1511] Processing CreateTablet for tablet f18edd9a005041c18b857bd860ab40a5 (DEFAULT_TABLE table=TestTable [id=7c4f6a62acb54a5480a7374832a9b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:26.832343 23859 tablet_service.cc:1511] Processing CreateTablet for tablet f18edd9a005041c18b857bd860ab40a5 (DEFAULT_TABLE table=TestTable [id=7c4f6a62acb54a5480a7374832a9b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:26.832348 23711 tablet_service.cc:1511] Processing CreateTablet for tablet f18edd9a005041c18b857bd860ab40a5 (DEFAULT_TABLE table=TestTable [id=7c4f6a62acb54a5480a7374832a9b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:26.833349 23561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18edd9a005041c18b857bd860ab40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:26.833384 23859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18edd9a005041c18b857bd860ab40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:26.833384 23711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18edd9a005041c18b857bd860ab40a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:26.838749 23952 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Bootstrap starting.
I20260812 08:00:26.839744 23953 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Bootstrap starting.
I20260812 08:00:26.839908 23954 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Bootstrap starting.
I20260812 08:00:26.841559 23952 tablet_bootstrap.cc:654] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:26.842005 23954 tablet_bootstrap.cc:654] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:26.842005 23953 tablet_bootstrap.cc:654] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:26.842382 23952 log.cc:826] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:26.842876 23954 log.cc:826] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:26.842876 23953 log.cc:826] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:26.844230 23952 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: No bootstrap required, opened a new log
I20260812 08:00:26.844455 23952 ts_tablet_manager.cc:1403] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 08:00:26.844532 23953 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: No bootstrap required, opened a new log
I20260812 08:00:26.844532 23954 tablet_bootstrap.cc:492] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: No bootstrap required, opened a new log
I20260812 08:00:26.844722 23954 ts_tablet_manager.cc:1403] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20260812 08:00:26.844722 23953 ts_tablet_manager.cc:1403] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:00:26.847361 23952 raft_consensus.cc:359] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.847577 23952 raft_consensus.cc:385] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:26.847647 23952 raft_consensus.cc:740] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db50616c97e4a9c9c298ed2df692dc1, State: Initialized, Role: FOLLOWER
I20260812 08:00:26.847836 23954 raft_consensus.cc:359] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.847836 23953 raft_consensus.cc:359] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.848028 23954 raft_consensus.cc:385] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:26.848028 23953 raft_consensus.cc:385] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:26.848090 23953 raft_consensus.cc:740] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86c753e99aa42f5a61e8cff2e2581d6, State: Initialized, Role: FOLLOWER
I20260812 08:00:26.848090 23954 raft_consensus.cc:740] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1a83fcf0fb4125a1765ee32d812593, State: Initialized, Role: FOLLOWER
I20260812 08:00:26.848222 23952 consensus_queue.cc:260] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.848485 23953 consensus_queue.cc:260] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.848567 23954 consensus_queue.cc:260] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:26.849211 23952 ts_tablet_manager.cc:1434] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I20260812 08:00:26.849208 23788 heartbeater.cc:499] Master 127.22.208.190:35703 was elected leader, sending a full tablet report...
I20260812 08:00:26.849339 23953 ts_tablet_manager.cc:1434] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.005s
I20260812 08:00:26.849553 23636 heartbeater.cc:499] Master 127.22.208.190:35703 was elected leader, sending a full tablet report...
I20260812 08:00:26.849869 23932 heartbeater.cc:499] Master 127.22.208.190:35703 was elected leader, sending a full tablet report...
I20260812 08:00:26.850044 23954 ts_tablet_manager.cc:1434] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
W20260812 08:00:26.870950 23637 tablet.cc:2406] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:27.019060 23961 raft_consensus.cc:493] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:27.019449 23961 raft_consensus.cc:515] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:27.020808 23961 leader_election.cc:290] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db50616c97e4a9c9c298ed2df692dc1 (127.22.208.129:40449), bf1a83fcf0fb4125a1765ee32d812593 (127.22.208.131:39731)
W20260812 08:00:27.023464 23933 tablet.cc:2406] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:27.026394 23582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f18edd9a005041c18b857bd860ab40a5" candidate_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db50616c97e4a9c9c298ed2df692dc1" is_pre_election: true
I20260812 08:00:27.026728 23882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f18edd9a005041c18b857bd860ab40a5" candidate_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1a83fcf0fb4125a1765ee32d812593" is_pre_election: true
I20260812 08:00:27.026791 23582 raft_consensus.cc:2468] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86c753e99aa42f5a61e8cff2e2581d6 in term 0.
I20260812 08:00:27.027053 23882 raft_consensus.cc:2468] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86c753e99aa42f5a61e8cff2e2581d6 in term 0.
I20260812 08:00:27.027417 23654 leader_election.cc:304] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db50616c97e4a9c9c298ed2df692dc1, a86c753e99aa42f5a61e8cff2e2581d6; no voters: 
I20260812 08:00:27.027724 23961 raft_consensus.cc:2804] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:00:27.027805 23961 raft_consensus.cc:493] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:27.027840 23961 raft_consensus.cc:3060] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:27.028964 23961 raft_consensus.cc:515] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:27.029452 23961 leader_election.cc:290] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [CANDIDATE]: Term 1 election: Requested vote from peers 0db50616c97e4a9c9c298ed2df692dc1 (127.22.208.129:40449), bf1a83fcf0fb4125a1765ee32d812593 (127.22.208.131:39731)
I20260812 08:00:27.029915 23882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f18edd9a005041c18b857bd860ab40a5" candidate_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1a83fcf0fb4125a1765ee32d812593"
I20260812 08:00:27.029999 23582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f18edd9a005041c18b857bd860ab40a5" candidate_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db50616c97e4a9c9c298ed2df692dc1"
I20260812 08:00:27.030098 23882 raft_consensus.cc:3060] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:27.030164 23582 raft_consensus.cc:3060] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:27.031683 23882 raft_consensus.cc:2468] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86c753e99aa42f5a61e8cff2e2581d6 in term 1.
I20260812 08:00:27.032210 23655 leader_election.cc:304] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a86c753e99aa42f5a61e8cff2e2581d6, bf1a83fcf0fb4125a1765ee32d812593; no voters: 
I20260812 08:00:27.032241 23582 raft_consensus.cc:2468] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86c753e99aa42f5a61e8cff2e2581d6 in term 1.
I20260812 08:00:27.032473 23961 raft_consensus.cc:2804] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:27.032735 23961 raft_consensus.cc:697] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [term 1 LEADER]: Becoming Leader. State: Replica: a86c753e99aa42f5a61e8cff2e2581d6, State: Running, Role: LEADER
I20260812 08:00:27.033097 23961 consensus_queue.cc:237] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } }
I20260812 08:00:27.036928 23411 catalog_manager.cc:5719] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a86c753e99aa42f5a61e8cff2e2581d6 (127.22.208.130). New cstate: current_term: 1 leader_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a86c753e99aa42f5a61e8cff2e2581d6" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 39217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:27.065456 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:27.067710 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0db50616c97e4a9c9c298ed2df692dc1 to finish bootstrapping
I20260812 08:00:27.074975 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a86c753e99aa42f5a61e8cff2e2581d6 to finish bootstrapping
W20260812 08:00:27.078882 23789 tablet.cc:2406] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:27.081558 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bf1a83fcf0fb4125a1765ee32d812593 to finish bootstrapping
I20260812 08:00:27.108392 23882 raft_consensus.cc:1275] T f18edd9a005041c18b857bd860ab40a5 P bf1a83fcf0fb4125a1765ee32d812593 [term 1 FOLLOWER]: Refusing update from remote peer a86c753e99aa42f5a61e8cff2e2581d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:27.108392 23582 raft_consensus.cc:1275] T f18edd9a005041c18b857bd860ab40a5 P 0db50616c97e4a9c9c298ed2df692dc1 [term 1 FOLLOWER]: Refusing update from remote peer a86c753e99aa42f5a61e8cff2e2581d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:27.109455 23964 consensus_queue.cc:1048] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1a83fcf0fb4125a1765ee32d812593" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:27.109687 23961 consensus_queue.cc:1048] T f18edd9a005041c18b857bd860ab40a5 P a86c753e99aa42f5a61e8cff2e2581d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db50616c97e4a9c9c298ed2df692dc1" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:27.119195 23970 mvcc.cc:204] Tried to move back new op lower bound from 7317592584627204096 to 7317592584329900032. Current Snapshot: MvccSnapshot[applied={T|T < 7317592584627204096}]
I20260812 08:00:27.474026 23711 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260812 08:00:27.474027 23859 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:27.474105 23561 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 7ea33df98d1f4b19814a4389ae0fe3d9 | 127.22.208.190:35703 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                            Value                                                                                             |      Tags       |         Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                                          | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                                          | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                                         | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                            | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                                         | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                         | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                           | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                                          | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:37627 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 0db50616c97e4a9c9c298ed2df692dc1 | 127.22.208.129:40449 | HEALTHY | <none>   |       0        |       0
 a86c753e99aa42f5a61e8cff2e2581d6 | 127.22.208.130:39217 | HEALTHY | <none>   |       1        |       0
 bf1a83fcf0fb4125a1765ee32d812593 | 127.22.208.131:39731 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
               Flag               |                                                                                          Value                                                                                           |      Tags       |      Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_server_key_size             | 768                                                                                                                                                                                      | experimental    | all 3 server(s) checked
 local_ip_for_outbound_sockets    | 127.22.208.129                                                                                                                                                                           | experimental    | 127.22.208.129:40449
 local_ip_for_outbound_sockets    | 127.22.208.130                                                                                                                                                                           | experimental    | 127.22.208.130:39217
 local_ip_for_outbound_sockets    | 127.22.208.131                                                                                                                                                                           | experimental    | 127.22.208.131:39731
 never_fsync                      | true                                                                                                                                                                                     | unsafe,advanced | all 3 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                        | unsafe,hidden   | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                     | unsafe          | all 3 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                       | hidden          | all 3 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden          | 127.22.208.129:40449
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden          | 127.22.208.130:39217
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestInsertAndMutateThroughConsensus.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden          | 127.22.208.131:39731

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:37627 | all 3 server(s) checked
 time_source         | builtin              | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
   Name    | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 08:00:27.502954 23362 log_verifier.cc:126] Checking tablet f18edd9a005041c18b857bd860ab40a5
I20260812 08:00:27.541414 23362 log_verifier.cc:177] Verified matching terms for 51 ops in tablet f18edd9a005041c18b857bd860ab40a5
I20260812 08:00:27.563540 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 23486
I20260812 08:00:27.572404 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 23639
I20260812 08:00:27.583762 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 23791
I20260812 08:00:27.595182 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 23377
2026-08-12T08:00:27Z chronyd exiting
[       OK ] RaftConsensusITest.TestInsertAndMutateThroughConsensus (1777 ms)
[ RUN      ] RaftConsensusITest.InsertAndDeleteWithCrashyNodes
I20260812 08:00:27.624625 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:00:27.624706 23362 ts_itest-base.cc:116] --------------
I20260812 08:00:27.624725 23362 ts_itest-base.cc:117] 7 tablet servers
I20260812 08:00:27.624744 23362 ts_itest-base.cc:118] 7 replicas per TS
I20260812 08:00:27.624761 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:00:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:00:27Z Disabled control of system clock
I20260812 08:00:27.635957 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:41383
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:41383 with env {}
W20260812 08:00:27.756422 24024 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:27.756733 24024 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:27.756821 24024 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:27.761004 24024 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:00:27.761118 24024 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:27.761162 24024 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:00:27.761206 24024 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:00:27.765897 24024 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:41383
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.24024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:27.767256 24024 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:27.768532 24024 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:27.774901 24033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:27.774901 24030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:27.774986 24031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:27.775189 24024 server_base.cc:1061] running on GCE node
I20260812 08:00:27.775806 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:27.776789 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:27.778005 24024 hybrid_clock.cc:648] HybridClock initialized: now 1786521627777992 us; error 31 us; skew 500 ppm
I20260812 08:00:27.780143 24024 webserver.cc:533] Webserver started at http://127.22.208.190:37905/ using document root <none> and password file <none>
I20260812 08:00:27.780733 24024 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:27.780809 24024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:27.780987 24024 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:27.783499 24024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/instance:
uuid: "10db3a93a2c74676a9fc532fdd526733"
format_stamp: "Formatted at 2026-08-12 08:00:27 on dist-test-slave-dhph"
I20260812 08:00:27.784020 24024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal/instance:
uuid: "10db3a93a2c74676a9fc532fdd526733"
format_stamp: "Formatted at 2026-08-12 08:00:27 on dist-test-slave-dhph"
I20260812 08:00:27.788022 24024 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:00:27.790845 24039 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:27.791988 24024 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:27.792109 24024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
uuid: "10db3a93a2c74676a9fc532fdd526733"
format_stamp: "Formatted at 2026-08-12 08:00:27 on dist-test-slave-dhph"
I20260812 08:00:27.792204 24024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:27.828984 24024 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:27.829824 24024 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:27.830154 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:27.839078 24024 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:41383
I20260812 08:00:27.839097 24101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:41383 every 8 connection(s)
I20260812 08:00:27.840225 24024 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb
I20260812 08:00:27.843201 24102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:27.844301 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24024
I20260812 08:00:27.844416 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/wal/instance
I20260812 08:00:27.849839 24102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733: Bootstrap starting.
I20260812 08:00:27.852524 24102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:27.853307 24102 log.cc:826] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:27.855075 24102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733: No bootstrap required, opened a new log
I20260812 08:00:27.857599 24102 raft_consensus.cc:359] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } }
I20260812 08:00:27.857847 24102 raft_consensus.cc:385] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:27.857930 24102 raft_consensus.cc:740] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10db3a93a2c74676a9fc532fdd526733, State: Initialized, Role: FOLLOWER
I20260812 08:00:27.858356 24102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } }
I20260812 08:00:27.858505 24102 raft_consensus.cc:399] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:00:27.858610 24102 raft_consensus.cc:493] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:00:27.858749 24102 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:27.859623 24102 raft_consensus.cc:515] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } }
I20260812 08:00:27.859992 24102 leader_election.cc:304] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10db3a93a2c74676a9fc532fdd526733; no voters: 
I20260812 08:00:27.860353 24102 leader_election.cc:290] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:00:27.860472 24108 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:27.860646 24108 raft_consensus.cc:697] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [term 1 LEADER]: Becoming Leader. State: Replica: 10db3a93a2c74676a9fc532fdd526733, State: Running, Role: LEADER
I20260812 08:00:27.860944 24108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } }
I20260812 08:00:27.861846 24102 sys_catalog.cc:565] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:00:27.862877 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10db3a93a2c74676a9fc532fdd526733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } } }
I20260812 08:00:27.862890 24109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10db3a93a2c74676a9fc532fdd526733. Latest consensus state: current_term: 1 leader_uuid: "10db3a93a2c74676a9fc532fdd526733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10db3a93a2c74676a9fc532fdd526733" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 41383 } } }
I20260812 08:00:27.863118 24109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:27.863430 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733 [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:27.864006 24118 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:00:27.867621 24118 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:00:27.873947 24118 catalog_manager.cc:1383] Generated new cluster ID: 4cc300b664a14248a533a64d22586ebb
I20260812 08:00:27.874065 24118 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:00:27.887398 24118 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:00:27.888373 24118 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:00:27.894603 24118 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 10db3a93a2c74676a9fc532fdd526733: Generated new TSK 0
I20260812 08:00:27.895291 24118 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:00:27.898833 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:28.020538 24131 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:28.021004 24131 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:28.021092 24131 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:28.021132 24131 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:28.021255 24131 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:28.021304 24131 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:28.021422 24131 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:28.025902 24131 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:28.026014 24131 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:28.026041 24131 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:28.026203 24131 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:00:28.031153 24131 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:28.032416 24131 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:28.033739 24131 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:28.037051 24131 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:28.041314 24140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.041275 24137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.041292 24138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:28.041828 24131 server_base.cc:1061] running on GCE node
I20260812 08:00:28.042304 24131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:28.042972 24131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:28.044144 24131 hybrid_clock.cc:648] HybridClock initialized: now 1786521628044123 us; error 36 us; skew 500 ppm
I20260812 08:00:28.046532 24131 webserver.cc:533] Webserver started at http://127.22.208.129:37489/ using document root <none> and password file <none>
I20260812 08:00:28.047443 24131 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:28.047616 24131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:28.047887 24131 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:28.050153 24131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/instance:
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.050892 24131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal/instance:
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.055024 24131 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20260812 08:00:28.057639 24146 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.058908 24131 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:00:28.059104 24131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.059237 24131 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:28.084784 24131 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:28.085594 24131 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:28.085843 24131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:28.086621 24131 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:28.087905 24131 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:28.087970 24131 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.088055 24131 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:28.088110 24131 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.099885 24131 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:40873
I20260812 08:00:28.099900 24280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:40873 every 8 connection(s)
I20260812 08:00:28.101060 24131 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
I20260812 08:00:28.107055 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24131
I20260812 08:00:28.107182 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal/instance
I20260812 08:00:28.110128 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:28.112928 24282 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:28.113222 24282 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:28.113868 24282 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:28.116043 24058 ts_manager.cc:194] Registered new tserver with Master: f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873)
I20260812 08:00:28.117933 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:43127
I20260812 08:00:28.130908 24282 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:28.229321 24286 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:28.229563 24286 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:28.229583 24286 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:28.229598 24286 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:28.229863 24286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:28.229928 24286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:28.230027 24286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:28.234079 24286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:28.234246 24286 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:28.234299 24286 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:28.234426 24286 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:00:28.239282 24286 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:28.240693 24286 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:28.241858 24286 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:28.244876 24286 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:28.249766 24291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.249879 24295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.249864 24292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:28.250545 24286 server_base.cc:1061] running on GCE node
I20260812 08:00:28.251147 24286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:28.251916 24286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:28.253444 24286 hybrid_clock.cc:648] HybridClock initialized: now 1786521628253237 us; error 69 us; skew 500 ppm
I20260812 08:00:28.255925 24286 webserver.cc:533] Webserver started at http://127.22.208.130:33433/ using document root <none> and password file <none>
I20260812 08:00:28.256579 24286 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:28.256675 24286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:28.256916 24286 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:28.259203 24286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/instance:
uuid: "ada36b19ad0e4386b68527e1ab0ce923"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.259912 24286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal/instance:
uuid: "ada36b19ad0e4386b68527e1ab0ce923"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.265131 24286 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:00:28.268340 24301 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.269649 24286 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260812 08:00:28.269778 24286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "ada36b19ad0e4386b68527e1ab0ce923"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.269919 24286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:28.293275 24286 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:28.294040 24286 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:28.294266 24286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:28.294986 24286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:28.296099 24286 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:28.296154 24286 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.296227 24286 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:28.296270 24286 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.307603 24286 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:36403
I20260812 08:00:28.307623 24428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:36403 every 8 connection(s)
I20260812 08:00:28.308719 24286 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
I20260812 08:00:28.309037 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24286
I20260812 08:00:28.309195 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal/instance
I20260812 08:00:28.312134 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:28.320089 24429 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:28.320431 24429 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:28.321200 24429 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:28.322789 24058 ts_manager.cc:194] Registered new tserver with Master: ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403)
I20260812 08:00:28.323594 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:51833
I20260812 08:00:28.336236 24429 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:28.434190 24433 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:28.434449 24433 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:28.434470 24433 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:28.434487 24433 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:28.434531 24433 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:28.434700 24433 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:28.434779 24433 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:28.438674 24433 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:28.438764 24433 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:28.438783 24433 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:28.438843 24433 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:00:28.443818 24433 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:28.445025 24433 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:28.446163 24433 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:28.449190 24433 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:28.453557 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.453596 24438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.453889 24442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:28.454530 24433 server_base.cc:1061] running on GCE node
I20260812 08:00:28.454977 24433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:28.455590 24433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:28.456770 24433 hybrid_clock.cc:648] HybridClock initialized: now 1786521628456747 us; error 35 us; skew 500 ppm
I20260812 08:00:28.458925 24433 webserver.cc:533] Webserver started at http://127.22.208.131:43791/ using document root <none> and password file <none>
I20260812 08:00:28.459579 24433 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:28.459668 24433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:28.459900 24433 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:28.461676 24433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/instance:
uuid: "9dede2273d3a4266a1bf7a5b368080f8"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.462217 24433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal/instance:
uuid: "9dede2273d3a4266a1bf7a5b368080f8"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.466754 24433 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:00:28.469175 24449 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.470260 24433 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:28.470412 24433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "9dede2273d3a4266a1bf7a5b368080f8"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.470525 24433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:28.488124 24433 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:28.489133 24433 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:28.489336 24433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:28.489959 24433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:28.491060 24433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:28.491110 24433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.491149 24433 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:28.491165 24433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.501659 24433 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:39533
I20260812 08:00:28.501686 24585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:39533 every 8 connection(s)
I20260812 08:00:28.502709 24433 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 08:00:28.511538 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24433
I20260812 08:00:28.511804 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal/instance
I20260812 08:00:28.513691 24586 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:28.513979 24586 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:28.514417 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:0
--local_ip_for_outbound_sockets=127.22.208.132
--webserver_interface=127.22.208.132
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:28.514592 24586 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:28.516021 24058 ts_manager.cc:194] Registered new tserver with Master: 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533)
I20260812 08:00:28.516937 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:46705
I20260812 08:00:28.529289 24586 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:28.639384 24589 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:28.639678 24589 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:28.639725 24589 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:28.639767 24589 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:28.639853 24589 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:28.639909 24589 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:28.640010 24589 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:28.644541 24589 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:28.644711 24589 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:28.644757 24589 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:28.644865 24589 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:00:28.650163 24589 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:28.651420 24589 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:28.653555 24589 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:28.656940 24589 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:28.661191 24599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.661295 24598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.661398 24601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:28.661899 24589 server_base.cc:1061] running on GCE node
I20260812 08:00:28.662433 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:28.663429 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:28.664614 24589 hybrid_clock.cc:648] HybridClock initialized: now 1786521628664576 us; error 58 us; skew 500 ppm
I20260812 08:00:28.667125 24589 webserver.cc:533] Webserver started at http://127.22.208.132:37633/ using document root <none> and password file <none>
I20260812 08:00:28.667788 24589 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:28.667853 24589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:28.668063 24589 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:28.669780 24589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/instance:
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.670327 24589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal/instance:
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.675443 24589 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20260812 08:00:28.678282 24607 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.679683 24589 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:28.679807 24589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.679975 24589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:28.701917 24589 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:28.702737 24589 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:28.702972 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:28.703677 24589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:28.704802 24589 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:28.704879 24589 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.704954 24589 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:28.705005 24589 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.716490 24589 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:35833
I20260812 08:00:28.716707 24743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:35833 every 8 connection(s)
I20260812 08:00:28.717640 24589 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 08:00:28.723938 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24589
I20260812 08:00:28.724104 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal/instance
I20260812 08:00:28.726833 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.133:0
--local_ip_for_outbound_sockets=127.22.208.133
--webserver_interface=127.22.208.133
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:28.729660 24746 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:28.729993 24746 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:28.730809 24746 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:28.732283 24058 ts_manager.cc:194] Registered new tserver with Master: e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:28.733001 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:35961
I20260812 08:00:28.746395 24746 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:28.864522 24751 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:28.864886 24751 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:28.864934 24751 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:28.864969 24751 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:28.865020 24751 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:28.865047 24751 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:28.865114 24751 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:28.869135 24751 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:28.869233 24751 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:28.869253 24751 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:28.869349 24751 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.133
I20260812 08:00:28.874204 24751 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.22.208.133
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.133

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:28.875866 24751 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:28.877246 24751 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:28.880362 24751 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:28.884683 24757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.884809 24756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:28.884681 24759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:28.885001 24751 server_base.cc:1061] running on GCE node
I20260812 08:00:28.885524 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:28.886691 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:28.887924 24751 hybrid_clock.cc:648] HybridClock initialized: now 1786521628887822 us; error 112 us; skew 500 ppm
I20260812 08:00:28.890219 24751 webserver.cc:533] Webserver started at http://127.22.208.133:38847/ using document root <none> and password file <none>
I20260812 08:00:28.890944 24751 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:28.891011 24751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:28.891191 24751 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:28.892927 24751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/instance:
uuid: "dbc6ed4943d5493caaf8b5feba124db1"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.893458 24751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal/instance:
uuid: "dbc6ed4943d5493caaf8b5feba124db1"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.897382 24751 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20260812 08:00:28.900904 24766 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.902262 24751 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:00:28.902424 24751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "dbc6ed4943d5493caaf8b5feba124db1"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:28.902549 24751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:28.914773 24751 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:28.915572 24751 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:28.915797 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:28.916438 24751 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:28.917543 24751 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:28.917621 24751 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.917711 24751 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:28.917763 24751 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:28.929505 24751 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.133:35369
I20260812 08:00:28.929522 24893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.133:35369 every 8 connection(s)
I20260812 08:00:28.930546 24751 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 08:00:28.936002 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24751
I20260812 08:00:28.936192 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal/instance
I20260812 08:00:28.938910 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:0
--local_ip_for_outbound_sockets=127.22.208.134
--webserver_interface=127.22.208.134
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:28.943027 24894 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:28.943320 24894 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:28.943868 24894 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:28.944949 24056 ts_manager.cc:194] Registered new tserver with Master: dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369)
I20260812 08:00:28.945770 24056 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.133:54563
I20260812 08:00:28.958792 24894 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:29.071642 24901 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:29.071914 24901 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:29.071938 24901 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:29.071959 24901 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:29.072053 24901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:29.072116 24901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:29.072235 24901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:29.076503 24901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:29.076629 24901 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:29.076653 24901 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:29.076771 24901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
I20260812 08:00:29.082921 24901 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.24901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:29.084233 24901 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:29.085583 24901 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:29.088816 24901 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:29.093853 24909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:29.093849 24907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:29.094084 24901 server_base.cc:1061] running on GCE node
W20260812 08:00:29.093855 24906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:29.094511 24901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:29.095280 24901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:29.096652 24901 hybrid_clock.cc:648] HybridClock initialized: now 1786521629096632 us; error 36 us; skew 500 ppm
I20260812 08:00:29.098907 24901 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:29.099568 24901 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:29.099665 24901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:29.099918 24901 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:29.102087 24901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/instance:
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.102893 24901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal/instance:
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.107791 24901 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260812 08:00:29.110543 24915 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.111868 24901 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:00:29.112038 24901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.112197 24901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:29.142028 24901 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:29.142808 24901 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:29.143009 24901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:29.143610 24901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:29.144646 24901 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:29.144724 24901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.144805 24901 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:29.144845 24901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.155565 24901 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:29.155612 25049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
I20260812 08:00:29.156603 24901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:29.160112 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 24901
I20260812 08:00:29.160251 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal/instance
I20260812 08:00:29.164942 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:0
--local_ip_for_outbound_sockets=127.22.208.135
--webserver_interface=127.22.208.135
--webserver_port=0
--tserver_master_addrs=127.22.208.190:41383
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:29.169970 25051 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:29.170331 25051 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:29.171003 25051 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:29.172564 24054 ts_manager.cc:194] Registered new tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:29.173259 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:43023
I20260812 08:00:29.185987 25051 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:29.301568 25056 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:29.301813 25056 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:29.301834 25056 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:29.301851 25056 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:29.301929 25056 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:29.301975 25056 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:29.302069 25056 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:29.305766 25056 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:29.305859 25056 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:29.305876 25056 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:29.305987 25056 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
I20260812 08:00:29.311583 25056 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.25056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:29.312747 25056 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:29.313822 25056 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:29.317271 25056 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:29.321856 25061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:29.323238 25062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:29.323753 25056 server_base.cc:1061] running on GCE node
W20260812 08:00:29.324101 25064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:29.324944 25056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:29.325770 25056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:29.326992 25056 hybrid_clock.cc:648] HybridClock initialized: now 1786521629326966 us; error 51 us; skew 500 ppm
I20260812 08:00:29.329327 25056 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
I20260812 08:00:29.329952 25056 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:29.330042 25056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:29.330284 25056 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:29.332657 25056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/instance:
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.333326 25056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal/instance:
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.338364 25056 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20260812 08:00:29.341507 25071 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.343237 25056 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:29.343441 25056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:29.343657 25056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:29.369226 25056 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:29.370189 25056 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:29.370442 25056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:29.371366 25056 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:29.372541 25056 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:29.372620 25056 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.372695 25056 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:29.372750 25056 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:29.385118 25056 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
I20260812 08:00:29.385201 25203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:29.386421 25056 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:29.396509 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 25056
I20260812 08:00:29.396664 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal/instance
I20260812 08:00:29.397786 25205 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:29.398077 25205 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:29.398727 25205 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:29.399802 24058 ts_manager.cc:194] Registered new tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
I20260812 08:00:29.400421 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:33343
I20260812 08:00:29.411877 23362 external_mini_cluster.cc:955] 7 TS(s) registered with all masters
I20260812 08:00:29.412554 25205 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:29.429443 24054 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60644:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260812 08:00:29.430975 24054 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20260812 08:00:29.454862 24506 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.455992 24506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.457919 24822 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.460132 24822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.460451 24355 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.461552 24355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.462154 24200 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.463168 24200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.469444 25126 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.470533 25126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.471975 25232 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap starting.
I20260812 08:00:29.476192 24972 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.476197 24668 tablet_service.cc:1511] Processing CreateTablet for tablet a12d50650402488683716909cc273d1e (DEFAULT_TABLE table=test-workload [id=35ef33a70eab4846a37b4010d8e5a103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:29.476941 25232 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.477360 24668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.477382 24972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12d50650402488683716909cc273d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:29.477982 25232 log.cc:826] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.486513 25237 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
I20260812 08:00:29.488763 25238 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Bootstrap starting.
I20260812 08:00:29.488914 25239 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap starting.
I20260812 08:00:29.489804 25237 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.490938 25237 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.491262 25240 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap starting.
I20260812 08:00:29.493121 25238 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.493319 25237 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: No bootstrap required, opened a new log
W20260812 08:00:29.493515 25237 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.493080 25239 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.495467 25239 log.cc:826] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.495810 25240 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.496665 25240 log.cc:826] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.497243 25236 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
I20260812 08:00:29.497764 25239 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: No bootstrap required, opened a new log
I20260812 08:00:29.498050 25239 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.003s
I20260812 08:00:29.498859 25232 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: No bootstrap required, opened a new log
I20260812 08:00:29.498848 25238 log.cc:826] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.499150 25232 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.000s
I20260812 08:00:29.500358 25236 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.501245 25236 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.502390 25238 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: No bootstrap required, opened a new log
W20260812 08:00:29.502663 25238 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 42ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.502837 25232 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.503146 25232 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.503217 25232 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada36b19ad0e4386b68527e1ab0ce923, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.503501 25239 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.503767 25239 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.503836 25239 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.504591 25239 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.505923 25239 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260812 08:00:29.504575 25232 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.508100 25240 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: No bootstrap required, opened a new log
W20260812 08:00:29.508469 25240 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.509598 25232 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260812 08:00:29.513610 25235 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap starting.
W20260812 08:00:29.514335 24749 tablet.cc:2406] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:00:29.514568 24431 tablet.cc:2406] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.517890 25235 tablet_bootstrap.cc:654] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:29.517963 25236 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: No bootstrap required, opened a new log
I20260812 08:00:29.518261 25236 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.000s
I20260812 08:00:29.519150 25235 log.cc:826] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:29.521237 25235 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: No bootstrap required, opened a new log
W20260812 08:00:29.521507 25235 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 12ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.521350 25236 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.522204 25236 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.522303 25236 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.523434 25236 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.525017 25236 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260812 08:00:29.530979 25206 tablet.cc:2406] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.533882 25237 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.048s	user 0.006s	sys 0.000s
I20260812 08:00:29.533936 25235 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.000s
I20260812 08:00:29.538285 25235 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.538285 25237 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.538658 25237 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.538658 25235 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.538733 25237 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.538733 25235 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.539386 25235 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.539389 25237 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.540612 25235 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.004s
I20260812 08:00:29.542167 25237 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
W20260812 08:00:29.542603 24284 tablet.cc:2406] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.544791 25240 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent bootstrapping tablet: real 0.054s	user 0.007s	sys 0.000s
I20260812 08:00:29.545006 25238 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Time spent bootstrapping tablet: real 0.056s	user 0.006s	sys 0.000s
I20260812 08:00:29.547652 25240 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.547928 25240 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.548000 25240 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc6ed4943d5493caaf8b5feba124db1, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.548477 25240 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.547993 25238 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.549515 25240 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.004s
W20260812 08:00:29.549566 25053 tablet.cc:2406] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.548995 25238 raft_consensus.cc:385] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:29.550547 25238 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dede2273d3a4266a1bf7a5b368080f8, State: Initialized, Role: FOLLOWER
I20260812 08:00:29.551538 25238 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.552898 25238 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.006s
W20260812 08:00:29.553597 24899 tablet.cc:2406] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.556747 25245 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:29.556988 25245 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.558650 25245 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:29.560443 25254 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:29.560698 25254 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:29.563434 24587 tablet.cc:2406] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:29.563753 25250 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:29.563970 25250 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.567385 25250 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:29.568775 25254 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:29.575891 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:29.576152 25149 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 0.
I20260812 08:00:29.577335 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:29.577616 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 0.
I20260812 08:00:29.578994 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:29.579187 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 0.
I20260812 08:00:29.579690 24689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:29.579870 24689 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 0.
I20260812 08:00:29.581588 25251 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:29.581779 25251 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.583922 25251 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:29.584158 25246 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:29.584352 24612 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 
I20260812 08:00:29.584450 24227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:29.584419 25246 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.584625 24227 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 0.
I20260812 08:00:29.584642 25245 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:00:29.584722 25245 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:29.584765 25245 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.586061 25245 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.586144 25246 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:29.586870 25245 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 1 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:29.587585 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:29.587920 24993 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 0.
I20260812 08:00:29.588263 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:29.588307 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:29.588421 24994 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.588421 24835 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.589828 24994 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.590193 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.590421 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:29.590615 25149 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.590512 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:29.591168 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.592428 25149 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.592480 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.593385 24612 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 
I20260812 08:00:29.593456 24227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:29.593595 24227 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 0.
I20260812 08:00:29.593619 25245 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:29.593856 25245 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 LEADER]: Becoming Leader. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Running, Role: LEADER
I20260812 08:00:29.593879 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:29.594015 24225 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.594235 25245 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:29.595252 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.596207 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:29.596449 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.599062 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:29.599313 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 0.
I20260812 08:00:29.601101 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:29.601361 24994 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.601850 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:29.602010 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 0.
I20260812 08:00:29.602560 25074 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:29.603008 25250 raft_consensus.cc:2764] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 08:00:29.603436 24056 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132). New cstate: current_term: 1 leader_uuid: "e3a0fdc906c74926946b47c61bd5cea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: HEALTHY } } }
I20260812 08:00:29.605809 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:29.606046 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.608517 24375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:29.608700 24375 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:29.610140 24375 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1.
I20260812 08:00:29.612709 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:29.612893 24376 raft_consensus.cc:2376] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 1. Re-sending same reply.
I20260812 08:00:29.613822 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:29.614076 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.615156 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:29.615353 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.617810 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:29.618052 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.622618 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:29.622859 24994 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.625895 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:29.626209 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.629539 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:29.629788 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.630357 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:29.630772 25254 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:00:29.631450 24689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:29.631951 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:29.632126 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.633149 24689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:29.634871 24689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:29.640602 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:29.640815 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:29.640945 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.641023 24994 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.641580 24150 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:29.641940 25251 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:00:29.642951 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:29.643180 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.643934 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:29.644136 24994 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.645120 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:29.645342 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.645864 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:29.646440 25246 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 08:00:29.648892 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:29.649173 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:29.653646 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:29.653914 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 1: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:29.674072 25302 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:29.674191 25302 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260812 08:00:29.674104 25311 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 54ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.675592 24994 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.676343 24225 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.677021 24835 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.677021 25245 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:29.677511 25302 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:29.677934 25302 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:29.679303 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.679538 25149 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.683207 25245 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:29.683475 25330 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:29.684875 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:00:29.686633 25245 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 08:00:29.692420 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 63ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.697795 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 26ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.698405 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 101ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.699836 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 23ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.702557 25349 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.727470 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 20ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.729416 25311 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 122ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.733296 25337 mvcc.cc:204] Tried to move back new op lower bound from 7317592595141136384 to 7317592594836291584. Current Snapshot: MvccSnapshot[applied={T|T < 7317592595141136384}]
I20260812 08:00:29.736331 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.737521 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.740305 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.760332 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.761334 25333 mvcc.cc:204] Tried to move back new op lower bound from 7317592595141136384 to 7317592594836291584. Current Snapshot: MvccSnapshot[applied={T|T < 7317592595141136384}]
W20260812 08:00:29.764057 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 17ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.764318 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 10ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.773758 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.797791 25342 mvcc.cc:204] Tried to move back new op lower bound from 7317592595181621248 to 7317592594836291584. Current Snapshot: MvccSnapshot[applied={T|T < 7317592595141136384}]
W20260812 08:00:29.803751 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 41ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.804378 25206 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:29.827612 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 28ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.835807 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.838884 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.102s	user 0.014s	sys 0.008s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":288,"dirs.queue_time_us":30111,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":3969,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4197,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":1,"thread_start_us":351,"threads_started":2}
I20260812 08:00:29.840688 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:29.842149 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20260812 08:00:29.842798 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:29.852492 25311 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.859301 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.119s	user 0.024s	sys 0.000s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":265,"dirs.queue_time_us":29859,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":11666,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":856,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":325,"threads_started":2}
I20260812 08:00:29.860709 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.880448 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.140s	user 0.020s	sys 0.000s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":32421,"dirs.queue_time_us":24313,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":7280,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":300,"threads_started":2}
W20260812 08:00:29.883013 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 5ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.883139 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:29.883806 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260812 08:00:29.884261 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:29.889956 25311 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 13ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.899349 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.031s	user 0.002s	sys 0.004s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4672,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:29.900516 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 408 bytes on disk
I20260812 08:00:29.904877 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.020s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:29.905891 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 245 bytes on disk
I20260812 08:00:29.901069 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.140s	user 0.021s	sys 0.003s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":28620,"dirs.queue_time_us":46456,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":751,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":1144,"threads_started":2}
I20260812 08:00:29.906441 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20260812 08:00:29.907598 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.908149 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:29.911597 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":147,"lbm_reads_lt_1ms":8}
I20260812 08:00:29.912279 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:29.907472 25076 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.103s	user 0.019s	sys 0.004s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":25645,"dirs.queue_time_us":14363,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":8798,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":296,"threads_started":2}
I20260812 08:00:29.917236 25206 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:29.934680 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 72ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.943192 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.169s	user 0.021s	sys 0.004s Metrics: {"bytes_written":4662,"cfile_init":1,"compiler_manager_pool.queue_time_us":15339,"dirs.queue_time_us":32885,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":2377,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1114,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":349,"threads_started":2}
I20260812 08:00:29.944765 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:29.945642 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20260812 08:00:29.946156 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:29.948963 25076 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.028s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1794,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11204,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:29.950091 25206 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
W20260812 08:00:29.958542 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 08:00:29.968056 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 67ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.970553 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.062s	user 0.019s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":8198,"dirs.run_cpu_time_us":513,"dirs.run_wall_time_us":3587,"drs_written":1,"lbm_read_time_us":835,"lbm_reads_lt_1ms":46,"lbm_write_time_us":19507,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":24,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":489,"threads_started":4}
I20260812 08:00:29.971508 24749 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:29.973464 24614 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
E20260812 08:00:29.976184 25330 fault_injection.cc:59] injecting fault for kudu (pid 24589): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:29.986940 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 6ms of latency in SegmentAllocator::Sync()
I20260812 08:00:29.996256 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.160s	user 0.028s	sys 0.000s Metrics: {"bytes_written":4797,"cfile_init":1,"compiler_manager_pool.queue_time_us":24886,"dirs.queue_time_us":25734,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":2227,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20442,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":8,"thread_start_us":385,"threads_started":2}
I20260812 08:00:29.997728 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:30.008919 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.164s	user 0.007s	sys 0.004s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":2436,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":995,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:30.010910 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 245 bytes on disk
W20260812 08:00:30.016702 25349 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 112ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.017109 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.018333 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:30.030431 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.115s	user 0.010s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1592,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5345,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:30.031962 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 408 bytes on disk
I20260812 08:00:30.032907 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":129,"lbm_reads_lt_1ms":8}
W20260812 08:00:30.033414 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 5ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.033476 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:30.037153 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.087s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1973,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:30.038340 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
W20260812 08:00:30.046151 24332 tablet_service.cc:3080] error setting up scanner b6a626b83ca84aa9ab3a306373754679 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.159s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
W20260812 08:00:30.049641 24332 tablet_service.cc:3080] error setting up scanner 0cd2642b41cc4282a8eb99c9364f212a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.162s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
W20260812 08:00:30.053356 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 51ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.059816 24332 tablet_service.cc:3080] error setting up scanner 57cddce92e8f4a889d5208a5332068bb with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.173s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
I20260812 08:00:30.058503 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.136s	user 0.020s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":15571,"dirs.run_cpu_time_us":647,"dirs.run_wall_time_us":3530,"drs_written":1,"lbm_read_time_us":1101,"lbm_reads_lt_1ms":46,"lbm_write_time_us":5026,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":403,"threads_started":5}
I20260812 08:00:30.060622 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.061970 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20260812 08:00:30.064453 24332 tablet_service.cc:3080] error setting up scanner 306934dec420480ebda8d975041a0356 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.177s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
W20260812 08:00:30.069581 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 156ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.069731 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.068s	user 0.008s	sys 0.000s Metrics: {"bytes_written":4691,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1843,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":666,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2}
I20260812 08:00:30.072934 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.679065
I20260812 08:00:30.088438 23362 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 08:00:30.088943 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:35833
--local_ip_for_outbound_sockets=127.22.208.132
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37633
--webserver_interface=127.22.208.132
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:30.094849 24332 tablet_service.cc:3080] error setting up scanner 45dd127e0db1471b8f0d665c5a7258ab with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.208s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
W20260812 08:00:30.097985 24332 tablet_service.cc:3080] error setting up scanner 882c4a39b51049918ab8ca7559f99fec with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.211s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
I20260812 08:00:30.105338 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.086s	user 0.012s	sys 0.008s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":4582,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1244,"drs_written":1,"lbm_read_time_us":1242,"lbm_reads_lt_1ms":46,"lbm_write_time_us":923,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":311,"threads_started":5}
I20260812 08:00:30.105867 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.107487 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.109858 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.076s	user 0.016s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":8630,"dirs.run_cpu_time_us":357,"dirs.run_wall_time_us":2136,"drs_written":1,"lbm_read_time_us":679,"lbm_reads_lt_1ms":46,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":900,"threads_started":4}
I20260812 08:00:30.110512 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.111744 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.133512 25076 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.177s	user 0.019s	sys 0.003s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":2624,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1557,"drs_written":1,"lbm_read_time_us":1056,"lbm_reads_lt_1ms":54,"lbm_write_time_us":11172,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":358,"threads_started":5}
I20260812 08:00:30.134249 25206 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.134909 25076 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20260812 08:00:30.140084 24332 tablet_service.cc:3080] error setting up scanner ade88b6439ef475ebd084ea5e2e59d94 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.253s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
W20260812 08:00:30.150890 24332 tablet_service.cc:3080] error setting up scanner d24ccec4768b4d86a327b1484d16ccce with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.264s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
I20260812 08:00:30.153017 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.078s	user 0.016s	sys 0.004s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":411,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1375,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":108,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1558,"drs_written":1,"lbm_read_time_us":1290,"lbm_reads_lt_1ms":54,"lbm_write_time_us":677,"lbm_writes_lt_1ms":23,"mutex_wait_us":210,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":332,"threads_started":5}
I20260812 08:00:30.156088 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.157598 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.172526 25251 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.172685 25251 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.173508 25251 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.174075 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.174245 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.174724 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.175566 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:30.176142 25253 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.176290 25253 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.176690 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.178062 24151 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:30.178184 25253 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.178426 25251 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 08:00:30.180540 25254 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.180683 25254 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.181435 25254 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.182782 25250 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.182940 25250 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.183568 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.183872 25250 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.184321 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.184466 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 1.
I20260812 08:00:30.185218 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.185348 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 1.
I20260812 08:00:30.186007 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.186836 24824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.186991 24824 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 1.
I20260812 08:00:30.187572 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.187704 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 1.
W20260812 08:00:30.187947 24767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.188200 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.188432 25246 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.188544 25246 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.188594 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:30.188740 25149 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 1.
I20260812 08:00:30.189299 25246 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.189563 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.189718 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 1.
I20260812 08:00:30.190344 24994 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.191360 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.191677 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.191823 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 1.
I20260812 08:00:30.192309 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.192471 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 1.
I20260812 08:00:30.193626 25075 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:30.193881 25250 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:00:30.193939 25250 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.193984 25250 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:30.194396 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.194543 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 1.
W20260812 08:00:30.194715 25072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.195262 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:30.196094 25250 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.196882 25250 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.197412 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:30.197546 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:30.198755 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 2.
I20260812 08:00:30.198820 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.198962 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 1.
I20260812 08:00:30.199505 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:30.199937 25246 raft_consensus.cc:2764] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260812 08:00:30.200292 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:30.200484 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.200785 25254 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:00:30.200870 25254 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.200917 25254 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:30.202404 25254 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.203023 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:30.203210 24225 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 1 FOLLOWER]: Advancing to term 2
W20260812 08:00:30.203409 24767 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.204468 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 2.
I20260812 08:00:30.205330 25252 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:30.205466 25252 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.205907 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.206058 24993 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 1.
I20260812 08:00:30.207609 25252 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.210757 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.210970 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.214733 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:30.214951 24835 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:30.216295 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 2.
I20260812 08:00:30.217002 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:30.217163 24993 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:30.217314 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:30.217494 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.218430 24993 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 2.
I20260812 08:00:30.218475 25075 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:30.218758 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:30.218930 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 2: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:30.219312 25254 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 election: Requested vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.220549 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:30.220737 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.221715 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:30.221910 25149 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.222223 25250 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Leader election won for term 2
I20260812 08:00:30.222492 25250 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 LEADER]: Becoming Leader. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Running, Role: LEADER
I20260812 08:00:30.223003 25250 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:30.225095 25349 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 37ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.225512 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:30.225699 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.226401 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:30.226572 24993 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.226505 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:30.226730 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.227223 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:30.228119 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.228312 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.228714 24767 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:30.229297 25253 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 08:00:30.229673 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.191s	user 0.018s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":1510,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1042,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":1864,"drs_written":1,"lbm_read_time_us":1005,"lbm_reads_lt_1ms":50,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"mutex_wait_us":183,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":10,"thread_start_us":430,"threads_started":5}
I20260812 08:00:30.230505 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.231148 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.231508 25254 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 08:00:30.231635 24916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.233145 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f reported cstate change: term changed from 1 to 2, leader changed from e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132) to 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135). New cstate: current_term: 2 leader_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:30.246721 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.246968 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.248145 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.248328 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:30.253922 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.254127 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 2: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
W20260812 08:00:30.259001 24147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.260162 24916 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.260286 24916 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:30.260732 25252 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 08:00:30.263060 24302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.264326 25149 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:30.265791 24450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.267257 25072 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.275486 25072 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.277930 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.278221 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.278218 24332 tablet_service.cc:3080] error setting up scanner a03c0356802b4903b5f2377528394552 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.391s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
W20260812 08:00:30.281179 24332 tablet_service.cc:3080] error setting up scanner 4f5a4b6ef5ea47e3b1a1b9baf61ab838 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.394s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
I20260812 08:00:30.285346 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.285548 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 2: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
W20260812 08:00:30.293337 24147 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.294262 24302 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.306200 25462 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:30.306296 25462 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:30.307144 24835 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 08:00:30.307835 25462 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.309152 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 49ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.314909 25072 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f -> Peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Couldn't send request to peer e3a0fdc906c74926946b47c61bd5cea5. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:30.318081 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 08:00:30.318850 25250 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260812 08:00:30.320760 24993 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 08:00:30.322220 25462 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.324456 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.328795 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 77ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.334363 24225 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 08:00:30.335243 25250 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
E20260812 08:00:30.335429 25250 fault_injection.cc:59] injecting fault for kudu (pid 25056): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:30.360224 24768 connection.cc:466] server connection from 127.22.208.135:55841 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:55841 (request call id 4) could send its response
W20260812 08:00:30.384017 24303 connection.cc:466] server connection from 127.22.208.135:34815 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:34815 (request call id 4) could send its response
W20260812 08:00:30.406400 24918 connection.cc:466] server connection from 127.22.208.135:34185 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:34185 (request call id 4) could send its response
W20260812 08:00:30.488790 24332 tablet_service.cc:3080] error setting up scanner c3b0879585c142c7ad32976a85316c24 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.165s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "0393b09412c7411985de84b7e3d977e2"
I20260812 08:00:30.547395 25251 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:30.547554 25251 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.548405 25251 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.550945 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.551142 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 2.
I20260812 08:00:30.551440 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.551831 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.552184 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
W20260812 08:00:30.562135 24147 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.562258 24147 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:30.564059 25251 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 08:00:30.564352 24151 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:30.593298 24332 tablet_service.cc:3080] error setting up scanner 18f2318a50944ca78377246e5dce218e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.269s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592596060131328 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "fa8ba87852d343e681835cd24134942d"
I20260812 08:00:30.610865 25253 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:30.611044 25253 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:30.611109 25411 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:30.611431 25411 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:30.611467 25411 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:30.611491 25411 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:30.611552 25411 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:30.611590 25411 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:30.611686 25411 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 08:00:30.611965 25253 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.614974 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.615176 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 2.
I20260812 08:00:30.616364 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
W20260812 08:00:30.618225 25411 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:30.618369 25411 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:30.618420 25411 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:30.618516 25411 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:00:30.618443 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.619047 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.619200 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 2.
W20260812 08:00:30.622963 24767 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.623422 24772 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:30.623497 24772 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:30.623839 25253 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260812 08:00:30.624083 25246 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:30.624183 25246 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.624842 25246 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.625648 25411 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:35833
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=37633
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.25411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:30.626781 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.626935 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 2.
I20260812 08:00:30.627177 24993 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:30.627382 25411 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:30.627516 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.627655 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 2.
I20260812 08:00:30.629102 25411 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:30.630065 25252 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:30.630184 25252 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.630919 25252 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.631829 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.632015 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 2.
E20260812 08:00:30.633709 25411 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 08:00:30.636188 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:30.636348 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 2.
W20260812 08:00:30.636626 24916 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.636960 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:30.637099 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 2.
I20260812 08:00:30.638828 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:30.639019 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 2.
W20260812 08:00:30.639881 25483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:30.640269 25482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:30.640516 25485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:30.640501 24918 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: e3a0fdc906c74926946b47c61bd5cea5
W20260812 08:00:30.641347 24305 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:30.641728 25411 server_base.cc:1061] running on GCE node
I20260812 08:00:30.642447 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260812 08:00:30.642704 24919 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:30.642818 24302 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:30.643062 25252 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:00:30.643150 25252 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:30.643221 25252 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.643343 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:30.644464 25252 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.646003 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:30.646085 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:30.646193 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.646318 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.647290 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 3.
I20260812 08:00:30.647507 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 3.
I20260812 08:00:30.648862 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:30.649025 24225 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.649902 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 3.
I20260812 08:00:30.650647 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:30.650867 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 2.
I20260812 08:00:30.651289 24919 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904; no voters: 
W20260812 08:00:30.651600 24919 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:30.651780 24305 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:30.652002 25246 raft_consensus.cc:2764] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260812 08:00:30.653456 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:30.653599 24835 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.654623 25252 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.654767 25252 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:00:30.654770 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 3.
I20260812 08:00:30.654865 25252 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 LEADER]: Becoming Leader. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Running, Role: LEADER
I20260812 08:00:30.655231 25252 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:30.657241 24916 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.657364 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 69ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.659349 24052 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b reported cstate change: term changed from 2 to 3, leader changed from 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135) to b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134). New cstate: current_term: 3 leader_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:30.675909 25411 hybrid_clock.cc:648] HybridClock initialized: now 1786521630675828 us; error 161 us; skew 500 ppm
I20260812 08:00:30.679226 25411 webserver.cc:533] Webserver started at http://127.22.208.132:37633/ using document root <none> and password file <none>
I20260812 08:00:30.680135 25411 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:30.680259 25411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:30.686796 25411 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.003s
I20260812 08:00:30.697634 25498 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:30.699153 25411 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.003s	sys 0.000s
I20260812 08:00:30.699335 25411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:30.700071 25411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 5
Total live bytes: 5162
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:30.718218 25411 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:30.719159 25411 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:30.719400 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:30.720187 25411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:30.721879 25506 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:30.731806 25411 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:30.731914 25411 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
I20260812 08:00:30.732080 25411 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:30.734230 25411 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:30.734308 25411 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 08:00:30.734812 25506 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap starting.
W20260812 08:00:30.748252 25488 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:30.748356 25488 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260812 08:00:30.748431 25208 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:30.750025 24916 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b -> Peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Couldn't send request to peer e3a0fdc906c74926946b47c61bd5cea5. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:30.750393 25411 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:35833
I20260812 08:00:30.750998 25506 log.cc:826] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:30.751974 25411 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 08:00:30.752351 25640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:35833 every 8 connection(s)
I20260812 08:00:30.754812 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 25411
I20260812 08:00:30.754969 23362 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 08:00:30.755411 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:33345
--local_ip_for_outbound_sockets=127.22.208.135
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37235
--webserver_interface=127.22.208.135
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:30.758116 24225 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260812 08:00:30.758806 25488 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.760854 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.762681 24482 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:30.767671 25506 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=10} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:30.768209 25506 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap complete.
I20260812 08:00:30.768644 25506 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent bootstrapping tablet: real 0.034s	user 0.024s	sys 0.000s
I20260812 08:00:30.770138 24835 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260812 08:00:30.770891 25488 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260812 08:00:30.773000 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260812 08:00:30.773638 25488 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.775954 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 161ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.778666 25643 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:30.778996 25643 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:30.779768 25643 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:30.780429 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260812 08:00:30.781332 25252 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.783211 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 33ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.783713 24919 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b -> Peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Couldn't send request to peer 3e1efe84be134f1bb72a09c4c17aa52f. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:30.784567 25506 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:30.785331 25506 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Initialized, Role: FOLLOWER
I20260812 08:00:30.786028 25506 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:30.786338 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 140ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.787278 25506 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260812 08:00:30.789937 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:30.791527 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":243,"lbm_reads_lt_1ms":4}
I20260812 08:00:30.794121 24051 ts_manager.cc:194] Re-registered known tserver with Master: e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:30.797616 24051 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:56565
W20260812 08:00:30.798914 24481 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.799348 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.799397 24482 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.799886 24479 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.800028 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.800375 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.800948 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:30.803853 25643 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:30.807722 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.807940 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.812146 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.823781 24331 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.824921 24330 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.825479 24329 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.827131 24327 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.827680 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.829439 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.830662 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.833241 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.837633 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.839584 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.848611 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.859735 24797 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.860812 24799 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.863497 24789 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.863624 24794 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.864012 24796 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.865556 24795 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.866618 24801 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.867262 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.866252 24792 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:30.866498 24788 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:30.884152 25564 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:00:30.890280 25564 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260812 08:00:30.891832 25488 consensus_queue.cc:1050] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 08:00:30.895943 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 96ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.900012 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.900789 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 107ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.906635 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 52ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.956362 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 76ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.956423 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 62ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.957999 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:30.958364 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:30.960537 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:30.965893 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 138ms of latency in SegmentAllocator::Sync()
I20260812 08:00:30.968058 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:30.976454 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 54ms of latency in SegmentAllocator::Sync()
W20260812 08:00:30.999822 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 87ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.005486 25646 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:31.005858 25646 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:31.005898 25646 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:31.005926 25646 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:31.005992 25646 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:31.006145 25646 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260812 08:00:31.006243 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.006320 25646 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:31.012801 25646 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:31.013042 25646 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:31.013208 25646 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:31.013320 25646 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
I20260812 08:00:31.017297 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.018177 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 42ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.020437 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.062s	user 0.000s	sys 0.003s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":360,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:31.021081 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 14ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.022086 25646 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:33345
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=37235
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.25646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:31.023721 25646 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:31.025324 25646 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:31.029247 25646 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 08:00:31.032555 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.032584 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 44ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.034278 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.076s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":195,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:31.034420 25694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:31.035665 25691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:31.035866 25692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:31.036124 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 106ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.037484 25646 server_base.cc:1061] running on GCE node
I20260812 08:00:31.038017 25646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:31.038851 25646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:31.042084 25646 hybrid_clock.cc:648] HybridClock initialized: now 1786521631041970 us; error 134 us; skew 500 ppm
W20260812 08:00:31.043316 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.043713 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.083s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.044131 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.044911 25646 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
I20260812 08:00:31.045735 25646 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:31.045855 25646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:31.053826 25646 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.003s	sys 0.000s
I20260812 08:00:31.053844 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.061388 25714 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 08:00:31.061481 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 70ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.063004 25646 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.003s	sys 0.000s
I20260812 08:00:31.063155 25646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:31.063735 25646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 5
Total live bytes: 5162
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:31.079130 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.030s	user 0.002s	sys 0.000s Metrics: {"bytes_written":228,"delete_count":8,"lbm_write_time_us":127,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.079604 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.080782 25646 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
W20260812 08:00:31.080852 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 1ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.081583 25646 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:31.081749 25646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:31.082458 25646 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:31.083287 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":105,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.083731 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.083820 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 55ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.087333 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 144ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.094552 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 95ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.095021 25727 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:31.099912 25646 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:31.100042 25646 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.016s	user 0.001s	sys 0.000s
I20260812 08:00:31.100332 25646 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:31.102407 25646 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:31.102483 25646 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 08:00:31.103034 25727 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
W20260812 08:00:31.104552 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 99ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.120353 25646 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
I20260812 08:00:31.122366 25646 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:31.127285 25727 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:31.128321 25856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:31.132272 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 25646
I20260812 08:00:31.135842 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.118s	user 0.004s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.136523 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.138247 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 57ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.139380 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 24ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.140902 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.087s	user 0.001s	sys 0.000s Metrics: {"bytes_written":212,"delete_count":6,"lbm_write_time_us":185,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.141651 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.142968 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 20ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.144074 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.111s	user 0.001s	sys 0.001s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.144728 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.145958 25727 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=11 ignored=10} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 08:00:31.146559 25727 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap complete.
W20260812 08:00:31.146838 25727 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 57ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.160924 25857 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:31.161293 25857 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:31.162182 25857 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:31.164220 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 111ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.164208 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 29ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.169535 24051 ts_manager.cc:194] Re-registered known tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
I20260812 08:00:31.174903 24051 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:58709
I20260812 08:00:31.187348 25857 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:31.192224 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.108s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4666,"cfile_init":1,"dirs.queue_time_us":3278,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1899,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":120,"threads_started":1}
I20260812 08:00:31.193049 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:31.193607 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20260812 08:00:31.194051 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 17ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.194159 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.194854 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.050s	user 0.000s	sys 0.002s Metrics: {"bytes_written":226,"delete_count":8,"lbm_write_time_us":119,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.195521 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.196991 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.060s	user 0.001s	sys 0.000s Metrics: {"bytes_written":242,"delete_count":9,"lbm_write_time_us":159,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.197568 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.200644 25334 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 78ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.204576 25727 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.102s	user 0.023s	sys 0.003s
W20260812 08:00:31.204694 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 41ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.204839 24306 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260812 08:00:31.206846 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.239s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":156,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.207461 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.208223 25727 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:31.208951 25727 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:31.210608 25727 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:31.210769 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 61ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.211395 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 93ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.211803 25727 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260812 08:00:31.219514 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:31.220535 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20260812 08:00:31.224522 25324 scanner-internal.cc:458] Time spent opening tablet: real 1.176s	user 0.004s	sys 0.000s
W20260812 08:00:31.224522 25323 scanner-internal.cc:458] Time spent opening tablet: real 1.180s	user 0.004s	sys 0.000s
I20260812 08:00:31.225375 25788 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 2 FOLLOWER]: Advancing to term 3
W20260812 08:00:31.231810 25503 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.220s	user 0.000s	sys 0.000s
W20260812 08:00:31.233806 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 97ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.234264 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.228s	user 0.006s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":349,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.234933 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.238086 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 66ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.243526 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.246086 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 123ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.274057 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.071s	user 0.007s	sys 0.001s Metrics: {"bytes_written":4727,"cfile_init":1,"dirs.queue_time_us":232,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":2366,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":4,"thread_start_us":160,"threads_started":1}
I20260812 08:00:31.274983 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 189 bytes on disk
I20260812 08:00:31.275609 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.276261 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.277762 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.136s	user 0.002s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":171,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.278398 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.281057 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.087s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4711,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1616,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":3}
I20260812 08:00:31.281813 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.284852 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.287621 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 85ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.289397 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.007s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4764,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1853,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":6}
I20260812 08:00:31.290074 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 390 bytes on disk
I20260812 08:00:31.290635 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":89,"lbm_reads_lt_1ms":8}
I20260812 08:00:31.291270 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.584729
I20260812 08:00:31.305697 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.071s	user 0.001s	sys 0.000s Metrics: {"bytes_written":226,"delete_count":8,"lbm_write_time_us":139,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.306216 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.306442 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.108s	user 0.004s	sys 0.002s Metrics: {"bytes_written":4727,"cfile_init":1,"dirs.queue_time_us":204,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":3764,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":4,"thread_start_us":123,"threads_started":1}
I20260812 08:00:31.308245 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.017s	user 0.016s	sys 0.000s Metrics: {"bytes_written":5156,"cfile_cache_hit":26,"cfile_cache_hit_bytes":1147,"cfile_cache_miss":58,"cfile_cache_miss_bytes":3119,"cfile_init":11,"delta_iterators_relevant":14,"dirs.queue_time_us":1545,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":834,"drs_written":1,"lbm_read_time_us":1276,"lbm_reads_lt_1ms":102,"lbm_write_time_us":728,"lbm_writes_lt_1ms":26,"num_input_rowsets":4,"peak_mem_usage":131072,"rows_written":20,"thread_start_us":771,"threads_started":7}
I20260812 08:00:31.308862 25053 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.309473 24920 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.311277 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 189 bytes on disk
I20260812 08:00:31.311787 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.312358 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:31.317291 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 51ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.319900 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.013s	user 0.010s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":2833,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":720,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":181,"threads_started":2}
I20260812 08:00:31.320822 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.944825
I20260812 08:00:31.322034 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.043s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":226,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1727,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":112,"threads_started":1}
I20260812 08:00:31.323284 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 255 bytes on disk
I20260812 08:00:31.324023 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.324457 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.943726
W20260812 08:00:31.332500 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.339052 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 24ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.343964 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.023s	user 0.014s	sys 0.004s Metrics: {"bytes_written":5156,"cfile_cache_hit":14,"cfile_cache_hit_bytes":840,"cfile_cache_miss":30,"cfile_cache_miss_bytes":2230,"cfile_init":8,"delta_iterators_relevant":8,"dirs.queue_time_us":925,"drs_written":1,"lbm_read_time_us":1010,"lbm_reads_lt_1ms":62,"lbm_write_time_us":1301,"lbm_writes_lt_1ms":26,"mutex_wait_us":27,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":20,"spinlock_wait_cycles":1792,"thread_start_us":371,"threads_started":5}
I20260812 08:00:31.344640 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.345312 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.367137 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.043s	user 0.014s	sys 0.001s Metrics: {"bytes_written":5156,"cfile_cache_hit":16,"cfile_cache_hit_bytes":906,"cfile_cache_miss":32,"cfile_cache_miss_bytes":2296,"cfile_init":7,"delta_iterators_relevant":10,"dirs.queue_time_us":780,"drs_written":1,"lbm_read_time_us":1274,"lbm_reads_lt_1ms":60,"lbm_write_time_us":586,"lbm_writes_lt_1ms":26,"mutex_wait_us":25,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":20,"spinlock_wait_cycles":3840,"thread_start_us":541,"threads_started":5}
I20260812 08:00:31.368114 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.370469 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.058s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4764,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":6}
I20260812 08:00:31.370882 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.127s	user 0.006s	sys 0.000s Metrics: {"bytes_written":250,"delete_count":10,"lbm_write_time_us":123,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.371678 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.372015 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.372052 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.579492
I20260812 08:00:31.373986 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.166s	user 0.002s	sys 0.000s Metrics: {"bytes_written":242,"delete_count":9,"lbm_write_time_us":107,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:31.374759 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.096s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4764,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":786,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":6}
I20260812 08:00:31.374923 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:31.375638 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.587669
W20260812 08:00:31.386889 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 104ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.395745 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 93ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.397656 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 19ms of latency in SegmentAllocator::Sync()
W20260812 08:00:31.405743 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 75ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.420218 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.044s	user 0.016s	sys 0.000s Metrics: {"bytes_written":5156,"cfile_cache_hit":19,"cfile_cache_hit_bytes":958,"cfile_cache_miss":43,"cfile_cache_miss_bytes":2584,"cfile_init":10,"delta_iterators_relevant":10,"dirs.queue_time_us":990,"drs_written":1,"lbm_read_time_us":1334,"lbm_reads_lt_1ms":83,"lbm_write_time_us":661,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":20,"thread_start_us":468,"threads_started":6}
I20260812 08:00:31.420871 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.424624 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260812 08:00:31.483424 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.111s	user 0.012s	sys 0.001s Metrics: {"bytes_written":4908,"cfile_init":1,"compiler_manager_pool.queue_time_us":288,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":964,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":137,"threads_started":1}
I20260812 08:00:31.484534 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.946000
I20260812 08:00:31.492036 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.110s	user 0.013s	sys 0.000s Metrics: {"bytes_written":5156,"cfile_cache_hit":49,"cfile_cache_hit_bytes":2453,"cfile_cache_miss":17,"cfile_cache_miss_bytes":1197,"cfile_init":3,"delta_iterators_relevant":12,"drs_written":1,"lbm_read_time_us":499,"lbm_reads_lt_1ms":29,"lbm_write_time_us":796,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":20,"thread_start_us":342,"threads_started":6}
I20260812 08:00:31.494642 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.119s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":5142,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":143,"threads_started":1}
I20260812 08:00:31.495658 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.944789
I20260812 08:00:31.496833 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.497660 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20260812 08:00:31.508980 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.516876 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.032s	user 0.009s	sys 0.009s Metrics: {"bytes_written":5156,"cfile_cache_hit":13,"cfile_cache_hit_bytes":807,"cfile_cache_miss":27,"cfile_cache_miss_bytes":2149,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":406,"drs_written":1,"lbm_read_time_us":980,"lbm_reads_lt_1ms":55,"lbm_write_time_us":2748,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":25,"mutex_wait_us":146,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":20}
I20260812 08:00:31.517537 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.520571 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20260812 08:00:31.524560 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 92ms of latency in SegmentAllocator::Sync()
I20260812 08:00:31.620638 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.120s	user 0.012s	sys 0.004s Metrics: {"bytes_written":5156,"cfile_cache_hit":14,"cfile_cache_hit_bytes":840,"cfile_cache_miss":30,"cfile_cache_miss_bytes":2238,"cfile_init":8,"delta_iterators_relevant":8,"dirs.queue_time_us":58,"drs_written":1,"lbm_read_time_us":1031,"lbm_reads_lt_1ms":62,"lbm_write_time_us":1121,"lbm_writes_lt_1ms":26,"mutex_wait_us":48,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":20,"spinlock_wait_cycles":39808,"thread_start_us":358,"threads_started":5}
I20260812 08:00:31.625049 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:31.627058 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":271,"lbm_reads_lt_1ms":4}
E20260812 08:00:31.955351 25488 fault_injection.cc:59] injecting fault for kudu (pid 24901): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:31.979480 23362 ts_itest-base.cc:575] TS 5 appears to have crashed. Restarting.
I20260812 08:00:31.980039 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:44711
--local_ip_for_outbound_sockets=127.22.208.134
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37749
--webserver_interface=127.22.208.134
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:32.066243 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:32.071036 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.078685 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.086002 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.087397 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.094029 24792 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.094866 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.100654 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.119609 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.119620 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.127779 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.128254 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.135030 25918 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:32.135412 25918 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:32.135478 25918 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:32.135520 25918 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:32.135599 25918 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:32.135651 25918 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:32.135744 25918 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:32.140281 25918 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:32.140378 25918 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:32.140396 25918 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:32.140530 25918 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
W20260812 08:00:32.144099 25510 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.144099 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.145149 25929 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.145300 25929 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.145357 25918 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:44711
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=37749
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.25918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:32.146714 25918 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:32.146894 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:32.146966 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:32.147047 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:32.148254 25918 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:32.149982 25929 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
E20260812 08:00:32.153035 25918 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 08:00:32.153682 25788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:32.154203 24772 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:32.154312 24772 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:32.154781 25929 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:00:32.156270 25559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:32.158129 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:32.158279 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:32.158416 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:32.158742 25918 server_base.cc:1061] running on GCE node
W20260812 08:00:32.159682 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.161020 25918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:32.161847 25918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:32.163030 25918 hybrid_clock.cc:648] HybridClock initialized: now 1786521632162976 us; error 89 us; skew 500 ppm
I20260812 08:00:32.163149 25942 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.163296 25942 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.165470 25918 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:32.166913 25918 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:32.167109 25918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:32.167482 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.168592 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.169497 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.170396 25942 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
W20260812 08:00:32.173113 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.174292 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.174351 25502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:32.174980 25788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:32.175151 25918 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.000s
W20260812 08:00:32.175657 25502 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:32.177424 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
W20260812 08:00:32.177520 24178 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.177549 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.181043 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:32.181214 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 3.
W20260812 08:00:32.181398 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.181502 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:32.181525 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.181903 25499 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:32.183684 25942 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:00:32.184944 25971 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:32.186079 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:32.186208 24178 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.186836 25918 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.003s	sys 0.000s
I20260812 08:00:32.186959 25918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:32.187510 25918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 6
Total live bytes: 5774
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:32.188719 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.190268 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.192190 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.194300 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.197889 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.198720 25976 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.198872 25976 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.200016 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:32.201049 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:32.201679 25976 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:32.201694 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:32.201831 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 3.
W20260812 08:00:32.204942 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.206434 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.207790 25788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:32.208597 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.209235 25559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:32.209441 25559 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 3.
W20260812 08:00:32.209978 24453 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:32.210069 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:32.210301 25976 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:00:32.219615 25918 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:32.220479 25918 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:32.220719 25918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:32.221963 25918 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 08:00:32.224383 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.224612 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.224788 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.226140 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.226385 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.226529 25984 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:32.226791 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.228310 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.230254 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.230792 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.232086 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.232533 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.232533 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.234139 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.234148 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.235469 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.236228 25918 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:32.236331 25918 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.000s
I20260812 08:00:32.236428 25918 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 08:00:32.236785 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.238933 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.239259 25918 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:32.239317 25918 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
W20260812 08:00:32.240541 25533 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.240494 25510 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.241906 25510 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.241919 25533 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.242213 25532 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.242879 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.243848 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.244247 25984 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
W20260812 08:00:32.245968 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.249248 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.249655 25535 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40740: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.249841 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:32.251622 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.254148 25918 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:32.254207 26119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
I20260812 08:00:32.254987 26017 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.255149 26017 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.255870 26022 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.255996 26022 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.256122 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:32.256225 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.256326 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 3.
I20260812 08:00:32.256649 26022 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:32.257038 24178 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.257620 24179 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52518: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.257936 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:32.258026 26029 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.258065 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 3.
I20260812 08:00:32.258186 26029 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:32.258734 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.258853 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:32.258975 24225 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 3.
I20260812 08:00:32.259788 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:32.259930 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 3.
I20260812 08:00:32.260277 24151 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:32.261351 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:32.261519 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 3.
I20260812 08:00:32.261662 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:32.261828 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 3.
I20260812 08:00:32.262823 26017 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:32.262946 26017 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:00:32.262992 26017 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.263031 26017 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.263489 26029 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:32.264084 24305 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:32.264385 26022 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260812 08:00:32.264487 26022 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:32.264374 26017 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.264607 26022 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.265389 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:32.265537 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.266729 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 4.
I20260812 08:00:32.267583 26022 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.268497 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:32.268648 24835 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.269090 24375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:32.269263 24375 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 4: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:32.269800 24835 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 4.
I20260812 08:00:32.267709 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:32.270318 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 4: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:32.270692 26022 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:32.271649 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:32.271749 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:32.271822 24528 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.271926 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.272408 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:32.272589 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.272924 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:32.273064 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.273644 26017 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:32.277397 25918 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:32.280385 25559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:32.280551 25559 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 3.
I20260812 08:00:32.280761 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:32.280845 25564 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.280994 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:32.281174 24835 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.282087 25564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 4.
I20260812 08:00:32.282459 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 25918
I20260812 08:00:32.282447 24225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:32.282650 24225 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.283012 25720 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:32.283320 26029 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.278057 25984 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:32.285853 26029 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260812 08:00:32.287518 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:32.287469 25788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:32.288697 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 4.
I20260812 08:00:32.290364 24151 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923
I20260812 08:00:32.290494 25788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:32.290701 25788 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.290834 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:32.290987 25787 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 4: Already voted for candidate f29b76cef775415dbd2c0997fadb3904 in this term.
I20260812 08:00:32.291072 24305 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:32.291393 26022 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260812 08:00:32.291654 26017 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Leader election won for term 4
I20260812 08:00:32.291709 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:32.291738 25559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:32.291879 26017 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 LEADER]: Becoming Leader. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Running, Role: LEADER
I20260812 08:00:32.291942 25564 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 4: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:32.292238 26017 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:32.294198 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 125ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.295935 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 reported cstate change: term changed from 3 to 4, leader changed from b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134) to f29b76cef775415dbd2c0997fadb3904 (127.22.208.129). New cstate: current_term: 4 leader_uuid: "f29b76cef775415dbd2c0997fadb3904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:32.297806 26061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:32.299983 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:32.300182 25564 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 4: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:32.300737 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:32.300793 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
W20260812 08:00:32.301167 25723 leader_election.cc:343] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:32.301362 24151 leader_election.cc:343] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:32.301638 24151 leader_election.cc:343] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 4 election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:32.302515 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:32.302515 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:32.303097 24305 leader_election.cc:343] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:32.303468 24305 leader_election.cc:343] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:32.315306 26143 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:32.315706 26143 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:32.316423 26143 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:32.319823 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.319919 24052 ts_manager.cc:194] Re-registered known tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:32.321424 24052 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:48811
W20260812 08:00:32.323123 24791 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.323484 24796 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.324070 24792 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.324571 24801 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.328485 24801 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.329432 24792 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.331476 24792 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56186: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.331897 26124 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:32.331960 26124 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:32.333140 25564 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20260812 08:00:32.333629 24835 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
W20260812 08:00:32.333685 24151 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 -> Peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Couldn't send request to peer b6d90cd6408d4dc3b47ebf57ca72e91b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:32.334048 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20260812 08:00:32.334296 26124 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260812 08:00:32.334833 25984 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=30} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:32.334794 26124 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260812 08:00:32.335338 25984 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap complete.
W20260812 08:00:32.335529 25984 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 26ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.336998 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 69ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.338063 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 150ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.338331 26143 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:32.339345 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20260812 08:00:32.339464 25787 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20260812 08:00:32.340400 26157 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260812 08:00:32.340544 26124 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260812 08:00:32.341152 26160 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
W20260812 08:00:32.343046 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 37ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.343372 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 53ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.344521 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 93ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.344831 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.347368 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.349880 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.354287 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.361944 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.362100 25984 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.118s	user 0.031s	sys 0.005s
W20260812 08:00:32.363480 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:32.365083 25984 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.365552 25984 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
I20260812 08:00:32.366147 25984 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:32.367195 25984 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
W20260812 08:00:32.369314 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:32.375383 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:32.376132 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 368 bytes on disk
I20260812 08:00:32.377270 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20260812 08:00:32.383026 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 6ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.391548 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 81ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.402266 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.420614 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 5ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.430068 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.436559 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 82ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.437156 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.442386 26165 mvcc.cc:204] Tried to move back new op lower bound from 7317592606047531008 to 7317592605875482624. Current Snapshot: MvccSnapshot[applied={T|T < 7317592606028800000}]
I20260812 08:00:32.442664 26056 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:32.448638 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.450716 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 25ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.461030 26183 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 41ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.463469 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 162ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.463994 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.466073 26182 mvcc.cc:204] Tried to move back new op lower bound from 7317592606308003840 to 7317592605875482624. Current Snapshot: MvccSnapshot[applied={T|T < 7317592606028800000}]
I20260812 08:00:32.468572 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.468883 26164 mvcc.cc:204] Tried to move back new op lower bound from 7317592606308003840 to 7317592605875482624. Current Snapshot: MvccSnapshot[applied={T|T < 7317592606028800000}]
W20260812 08:00:32.471589 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 100ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.476423 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 31ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.478645 26163 mvcc.cc:204] Tried to move back new op lower bound from 7317592606246219776 to 7317592605875482624. Current Snapshot: MvccSnapshot[applied={T|T < 7317592606028800000}]
I20260812 08:00:32.479043 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.488444 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.491758 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.000s	sys 0.002s Metrics: {"bytes_written":259,"delete_count":10,"lbm_write_time_us":156,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.492244 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:32.496230 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 11ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.496465 26162 mvcc.cc:204] Tried to move back new op lower bound from 7317592606032957440 to 7317592605875482624. Current Snapshot: MvccSnapshot[applied={T|T < 7317592606028800000}]
W20260812 08:00:32.500195 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 27ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.503391 26183 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 77ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.504968 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.507994 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 23ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.509279 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.045s	user 0.000s	sys 0.001s Metrics: {"bytes_written":249,"delete_count":9,"lbm_write_time_us":231,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.509491 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.017s	user 0.005s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":155,"cfile_cache_miss":2,"cfile_cache_miss_bytes":164,"cfile_init":1,"delete_count":20,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":79,"lbm_reads_lt_1ms":6,"lbm_write_time_us":132,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":147,"threads_started":1,"update_count":0}
I20260812 08:00:32.509763 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.510645 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.516089 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 76ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.520769 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.072s	user 0.000s	sys 0.002s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":146,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.521378 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.525396 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 13ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.528410 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 96ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.532761 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.023s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":120,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.537741 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.542979 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 3ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.547006 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.018s	user 0.001s	sys 0.001s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.547696 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.548153 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.010s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":9,"thread_start_us":153,"threads_started":1}
I20260812 08:00:32.548794 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.552542 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 44ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.553519 25868 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 25ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.572448 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.580407 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.031s	user 0.004s	sys 0.000s Metrics: {"bytes_written":4668,"cfile_init":1,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":114,"threads_started":1}
W20260812 08:00:32.581089 26183 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 51ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.581458 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.987138
I20260812 08:00:32.582845 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.113s	user 0.007s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.583405 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.593989 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.083s	user 0.003s	sys 0.002s Metrics: {"bytes_written":4727,"cfile_init":1,"dirs.queue_time_us":2875,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1420,"drs_written":1,"lbm_read_time_us":587,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":4,"thread_start_us":159,"threads_started":1}
I20260812 08:00:32.594748 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.595866 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.014s	user 0.012s	sys 0.002s Metrics: {"bytes_written":5416,"cfile_cache_hit":23,"cfile_cache_hit_bytes":1512,"cfile_cache_miss":43,"cfile_cache_miss_bytes":3226,"cfile_init":11,"delta_iterators_relevant":12,"drs_written":1,"lbm_read_time_us":1471,"lbm_reads_lt_1ms":87,"lbm_write_time_us":567,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":30,"thread_start_us":384,"threads_started":5}
I20260812 08:00:32.597420 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.597918 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.599670 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.120s	user 0.002s	sys 0.000s Metrics: {"bytes_written":249,"delete_count":9,"lbm_write_time_us":127,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.600797 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.600713 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.053s	user 0.003s	sys 0.003s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":1776,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1085,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":470,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":134,"threads_started":1}
I20260812 08:00:32.601683 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 257 bytes on disk
I20260812 08:00:32.603094 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.603631 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.977593
I20260812 08:00:32.605489 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":134,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.605962 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.612300 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 93ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.613166 25347 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 127ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.615293 25346 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 6ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.625123 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.021s	user 0.011s	sys 0.003s Metrics: {"bytes_written":5416,"cfile_cache_hit":21,"cfile_cache_hit_bytes":1593,"cfile_cache_miss":27,"cfile_cache_miss_bytes":2559,"cfile_init":6,"delta_iterators_relevant":10,"dirs.queue_time_us":511,"drs_written":1,"lbm_read_time_us":804,"lbm_reads_lt_1ms":51,"lbm_write_time_us":903,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":30,"thread_start_us":792,"threads_started":5}
I20260812 08:00:32.625762 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.626324 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.121s	user 0.002s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":133,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.627142 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.627550 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.197s	user 0.002s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":158,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.628363 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.630548 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.632421 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.005s	user 0.001s	sys 0.000s Metrics: {"bytes_written":223,"delete_count":6,"lbm_write_time_us":132,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.632514 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"bytes_written":234,"delete_count":7,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.632926 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.632977 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:32.634063 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.050s	user 0.002s	sys 0.000s Metrics: {"bytes_written":234,"delete_count":8,"lbm_write_time_us":139,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:32.635099 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:32.640390 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 10ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.640727 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 236ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.651656 26183 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 52ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.652448 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.019s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":625,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":2337,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":140,"threads_started":1}
I20260812 08:00:32.653227 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.977531
W20260812 08:00:32.665380 25338 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 76ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.706419 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.069s	user 0.007s	sys 0.004s Metrics: {"bytes_written":4908,"cfile_init":1,"compiler_manager_pool.queue_time_us":2289,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":172,"threads_started":1}
I20260812 08:00:32.707108 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.100s	user 0.005s	sys 0.001s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":294,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":2023,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":178,"threads_started":1}
I20260812 08:00:32.707818 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 257 bytes on disk
I20260812 08:00:32.707837 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 257 bytes on disk
I20260812 08:00:32.708330 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.708618 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.708869 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.977443
I20260812 08:00:32.709162 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.977393
W20260812 08:00:32.718444 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 56ms of latency in SegmentAllocator::Sync()
I20260812 08:00:32.728521 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.019s	user 0.017s	sys 0.000s Metrics: {"bytes_written":5416,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1390,"cfile_cache_miss":30,"cfile_cache_miss_bytes":2730,"cfile_init":7,"delta_iterators_relevant":10,"dirs.queue_time_us":913,"drs_written":1,"lbm_read_time_us":950,"lbm_reads_lt_1ms":58,"lbm_write_time_us":651,"lbm_writes_lt_1ms":26,"mutex_wait_us":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":30,"spinlock_wait_cycles":8064,"thread_start_us":330,"threads_started":5}
I20260812 08:00:32.729231 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.729662 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.742311 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.147s	user 0.007s	sys 0.001s Metrics: {"bytes_written":4764,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1735,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7641,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":6}
I20260812 08:00:32.743333 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 400 bytes on disk
I20260812 08:00:32.743907 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":121,"lbm_reads_lt_1ms":8}
I20260812 08:00:32.745296 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.092s	user 0.014s	sys 0.000s Metrics: {"bytes_written":5416,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1390,"cfile_cache_miss":30,"cfile_cache_miss_bytes":2752,"cfile_init":8,"delta_iterators_relevant":10,"drs_written":1,"lbm_read_time_us":845,"lbm_reads_lt_1ms":62,"lbm_write_time_us":994,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":30,"thread_start_us":413,"threads_started":5}
I20260812 08:00:32.746763 24284 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.746961 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.790867
I20260812 08:00:32.747756 24152 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.765928 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.018s	user 0.013s	sys 0.000s Metrics: {"bytes_written":5416,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1165,"cfile_cache_miss":40,"cfile_cache_miss_bytes":3209,"cfile_init":8,"delta_iterators_relevant":8,"dirs.queue_time_us":1155,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":723,"drs_written":1,"lbm_read_time_us":1437,"lbm_reads_lt_1ms":72,"lbm_write_time_us":566,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":30,"thread_start_us":279,"threads_started":5}
I20260812 08:00:32.766763 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.767323 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.777735 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.068s	user 0.009s	sys 0.002s Metrics: {"bytes_written":5416,"cfile_cache_hit":26,"cfile_cache_hit_bytes":2200,"cfile_cache_miss":22,"cfile_cache_miss_bytes":1926,"cfile_init":5,"delta_iterators_relevant":10,"dirs.queue_time_us":1095,"drs_written":1,"lbm_read_time_us":431,"lbm_reads_lt_1ms":42,"lbm_write_time_us":1158,"lbm_writes_lt_1ms":26,"mutex_wait_us":24,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":30,"spinlock_wait_cycles":8704,"thread_start_us":391,"threads_started":5}
I20260812 08:00:32.778503 24899 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.779066 24773 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
E20260812 08:00:32.828424 26157 fault_injection.cc:59] injecting fault for kudu (pid 24131): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:32.848740 23362 ts_itest-base.cc:575] TS 0 appears to have crashed. Restarting.
I20260812 08:00:32.849255 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:40873
--local_ip_for_outbound_sockets=127.22.208.129
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37489
--webserver_interface=127.22.208.129
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:32.877112 25503 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.237s	user 0.000s	sys 0.000s
I20260812 08:00:32.879037 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.246s	user 0.004s	sys 0.004s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":257,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":2568,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":147,"threads_started":1}
I20260812 08:00:32.879880 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 257 bytes on disk
I20260812 08:00:32.880517 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20260812 08:00:32.881072 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.977531
W20260812 08:00:32.884166 24325 tablet_service.cc:3080] error setting up scanner 8ed6397d254e452e96a4f9344caa2a68 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.158s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
W20260812 08:00:32.884461 24328 tablet_service.cc:3080] error setting up scanner 3c0df84890984b93bddbf508dc0b22b4 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.158s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
W20260812 08:00:32.902380 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 91ms of latency in SegmentAllocator::Sync()
W20260812 08:00:32.904191 24328 tablet_service.cc:3080] error setting up scanner a515e7ff7d584578935bd853aed44faf with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.178s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
W20260812 08:00:32.904244 24325 tablet_service.cc:3080] error setting up scanner d620031748954395ae9ef875df785f1a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.178s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
W20260812 08:00:32.930626 24325 tablet_service.cc:3080] error setting up scanner d943dda3562f417784e41fc6fc2db961 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.204s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
W20260812 08:00:32.938742 24325 tablet_service.cc:3080] error setting up scanner 3fa121837a6a4a84b3067ab8c67370fb with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.212s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
W20260812 08:00:32.981264 24325 tablet_service.cc:3080] error setting up scanner f2f2271383da4075b7b74e0d8ebd6878 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.255s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
I20260812 08:00:32.998451 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.117s	user 0.015s	sys 0.000s Metrics: {"bytes_written":5416,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1390,"cfile_cache_miss":30,"cfile_cache_miss_bytes":2752,"cfile_init":7,"delta_iterators_relevant":10,"dirs.queue_time_us":1353,"drs_written":1,"lbm_read_time_us":946,"lbm_reads_lt_1ms":58,"lbm_write_time_us":734,"lbm_writes_lt_1ms":26,"mutex_wait_us":30,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":30,"thread_start_us":354,"threads_started":5}
I20260812 08:00:32.999276 25645 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:32.999897 25503 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20260812 08:00:33.007879 26248 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:33.008246 26248 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:33.008291 26248 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:33.008327 26248 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:33.008412 26248 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:33.008451 26248 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:33.008528 26248 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:33.014678 26248 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:33.014958 26248 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:33.015036 26248 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:33.015141 26248 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
W20260812 08:00:33.017323 24325 tablet_service.cc:3080] error setting up scanner 3cc4264ac36940b98003be5f006ea7be with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.291s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
I20260812 08:00:33.020649 26248 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:40873
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=37489
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.26248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:33.022178 26248 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:33.023836 26248 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:33.027287 26248 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 08:00:33.027508 26260 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.027751 26260 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.029017 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.029590 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.030011 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.030297 26260 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.030561 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.031175 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:33.031509 24302 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.031594 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:33.032214 26260 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 08:00:33.034871 26265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:33.035117 26248 server_base.cc:1061] running on GCE node
W20260812 08:00:33.034894 26268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:33.035261 26266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:33.035729 26248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:33.036546 26248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:33.037760 26248 hybrid_clock.cc:648] HybridClock initialized: now 1786521633037745 us; error 39 us; skew 500 ppm
I20260812 08:00:33.040071 26248 webserver.cc:533] Webserver started at http://127.22.208.129:37489/ using document root <none> and password file <none>
I20260812 08:00:33.040730 26248 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:33.040864 26248 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:33.046126 26274 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.046311 26274 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.047435 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.047470 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.047605 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 4.
I20260812 08:00:33.047791 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.047819 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.048242 26274 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:33.048400 25720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.049023 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:33.049439 25720 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.049533 25720 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923; no voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:33.049703 26274 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:00:33.050058 26248 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.002s	sys 0.000s
I20260812 08:00:33.056591 26282 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.056725 26282 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.057809 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.057979 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 4.
I20260812 08:00:33.058475 26282 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.059099 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:33.059310 26284 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:33.059746 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.059942 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 4.
I20260812 08:00:33.060362 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.060865 26248 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.003s	sys 0.000s
I20260812 08:00:33.061038 26248 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:33.061724 26248 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 6
Total live bytes: 6310
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:33.063735 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.068418 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.069043 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:33.069412 26282 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260812 08:00:33.080027 24325 tablet_service.cc:3080] error setting up scanner dfb09406615746eba1964c20ff4bd4ad with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.354s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
I20260812 08:00:33.081679 26293 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.081815 26293 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.082862 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.083014 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 4.
I20260812 08:00:33.083086 26293 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:33.083092 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.083254 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 4.
I20260812 08:00:33.083276 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.083436 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 4.
I20260812 08:00:33.084563 25502 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 
I20260812 08:00:33.084704 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.084780 26293 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260812 08:00:33.084939 26293 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.085044 26293 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:00:33.086625 26293 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:33.087374 25499 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.087673 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:33.087836 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:00:33.088678 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 5.
I20260812 08:00:33.089897 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:33.090065 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 4 FOLLOWER]: Advancing to term 5
W20260812 08:00:33.090997 25499 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.091351 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 5.
I20260812 08:00:33.092195 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:33.092355 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:00:33.092952 24835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:33.093511 26293 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:33.093770 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 5.
I20260812 08:00:33.094374 25501 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:33.094568 26293 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 08:00:33.094833 26293 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 LEADER]: Becoming Leader. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Running, Role: LEADER
I20260812 08:00:33.095188 26293 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 4.54, Last appended by leader: 54, Current term: 5, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.097795 26056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.097826 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
W20260812 08:00:33.099274 25660 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 52ms of latency in SegmentAllocator::Sync()
I20260812 08:00:33.100740 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 reported cstate change: term changed from 4 to 5, leader changed from f29b76cef775415dbd2c0997fadb3904 (127.22.208.129) to e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132). New cstate: current_term: 5 leader_uuid: "e3a0fdc906c74926946b47c61bd5cea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: HEALTHY } } }
I20260812 08:00:33.111706 26248 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:33.112524 26248 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:33.113008 26248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:33.113067 26302 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.113200 26302 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.113930 26248 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:33.113945 26302 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.114290 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.114476 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:33.115952 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:33.116617 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.116794 25787 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:33.117404 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.117589 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:33.118824 26307 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:33.120457 24772 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:33.120782 26302 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 4 FOLLOWER]: Advancing to term 5
W20260812 08:00:33.124261 24325 tablet_service.cc:3080] error setting up scanner 1bde618238bb4454bcfe44345713874e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.398s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
I20260812 08:00:33.125106 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.125283 26248 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:33.125353 26248 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
I20260812 08:00:33.125367 26057 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 4.
I20260812 08:00:33.125428 26248 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:33.125491 26309 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:33.125615 26309 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.129317 26302 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260812 08:00:33.130395 26248 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:33.130481 26248 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.000s	sys 0.003s
I20260812 08:00:33.130928 26307 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap starting.
W20260812 08:00:33.131070 24767 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.133797 26309 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:33.137399 25972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:33.156231 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.156476 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:33.156499 26248 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:40873
I20260812 08:00:33.158016 26248 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
I20260812 08:00:33.161535 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 26248
I20260812 08:00:33.164376 26456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:40873 every 8 connection(s)
I20260812 08:00:33.164546 26307 log.cc:826] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:33.175246 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.175467 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:33.176424 26293 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:33.176514 26293 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:33.177488 24835 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 4 index: 54. Preceding OpId from leader: term: 5 index: 55. (index mismatch)
I20260812 08:00:33.178088 26293 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260812 08:00:33.180624 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.180852 25787 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:33.181237 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 67ms of latency in SegmentAllocator::Sync()
I20260812 08:00:33.182188 26457 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:33.182446 26457 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:33.183187 26457 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:33.185439 25564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:33.185636 24052 ts_manager.cc:194] Re-registered known tserver with Master: f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873)
I20260812 08:00:33.186060 25972 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:33.186666 26309 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:00:33.186745 24052 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:60359
I20260812 08:00:33.188284 26309 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
E20260812 08:00:33.188694 26293 fault_injection.cc:59] injecting fault for kudu (pid 25411): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:33.199433 24824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.201828 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:33.206641 25972 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:33.209596 26307 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=50} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:33.210245 26307 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap complete.
W20260812 08:00:33.210541 26307 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 78ms of latency in SegmentAllocator::Sync()
I20260812 08:00:33.212957 23362 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 08:00:33.213357 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:35833
--local_ip_for_outbound_sockets=127.22.208.132
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37633
--webserver_interface=127.22.208.132
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:33.214058 26457 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:33.248960 24768 connection.cc:466] server connection from 127.22.208.132:38589 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.132:38589 (request call id 4) could send its response
I20260812 08:00:33.289434 26307 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent bootstrapping tablet: real 0.159s	user 0.041s	sys 0.013s
I20260812 08:00:33.292718 26307 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.297286 26307 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Initialized, Role: FOLLOWER
I20260812 08:00:33.298254 26307 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 4.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.299484 26307 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
W20260812 08:00:33.303086 24325 tablet_service.cc:3080] error setting up scanner 64c92d87623b42ea82fb5ff46966339c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.577s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
I20260812 08:00:33.306849 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:33.308013 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20260812 08:00:33.370361 24325 tablet_service.cc:3080] error setting up scanner dfa06081cf704851826b058af7cf2450 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.644s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
W20260812 08:00:33.375985 26470 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:33.376293 26470 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:33.376338 26470 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:33.376371 26470 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:33.376437 26470 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:33.376478 26470 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:33.376569 26470 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:33.380772 26470 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:33.380920 26470 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:33.380960 26470 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:33.381053 26470 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:00:33.386127 26470 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:35833
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=37633
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.26470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:33.387796 26470 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:33.389376 26470 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:33.392850 26470 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:33.397233 26478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:33.398535 26479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:33.399312 26470 server_base.cc:1061] running on GCE node
W20260812 08:00:33.399884 26481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:33.400403 26470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:33.401109 26470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:33.408653 26470 hybrid_clock.cc:648] HybridClock initialized: now 1786521633408621 us; error 72 us; skew 500 ppm
I20260812 08:00:33.411428 26470 webserver.cc:533] Webserver started at http://127.22.208.132:37633/ using document root <none> and password file <none>
I20260812 08:00:33.412281 26470 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:33.412408 26470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:33.417608 26470 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:33.428356 26491 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:33.429607 26470 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.001s	sys 0.000s
I20260812 08:00:33.429816 26470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:33.430279 26470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 6
Total live bytes: 6310
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:33.452544 26470 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:33.453408 26470 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:33.453617 26470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:33.454351 26470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:33.455744 26499 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:33.460162 26470 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:33.460249 26470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20260812 08:00:33.460316 26470 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:33.462284 26470 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:33.462394 26470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:33.462705 26499 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap starting.
I20260812 08:00:33.480906 26499 log.cc:826] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:33.481585 26302 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:33.481762 26302 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.482976 26302 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.483083 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.483237 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 5.
I20260812 08:00:33.483840 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.483987 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 5.
I20260812 08:00:33.484313 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.484464 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 5.
I20260812 08:00:33.485049 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.485196 26057 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 5.
I20260812 08:00:33.485576 24772 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1; no voters: 
W20260812 08:00:33.486748 24767 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:33.486929 26302 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260812 08:00:33.487013 26302 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:33.487079 26302 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.488303 26302 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.488770 26470 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:35833
I20260812 08:00:33.489356 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:33.489415 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:33.489491 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.489563 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.489545 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:33.489677 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.490620 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 6.
I20260812 08:00:33.490797 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 6.
I20260812 08:00:33.490819 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 6.
I20260812 08:00:33.492110 26470 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 08:00:33.492249 24772 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1; no voters: 
I20260812 08:00:33.495034 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:33.495159 26398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:33.495308 26402 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 4.
I20260812 08:00:33.494927 26302 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.495784 26302 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 FOLLOWER]: Leader election won for term 6
I20260812 08:00:33.495916 26302 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 LEADER]: Becoming Leader. State: Replica: dbc6ed4943d5493caaf8b5feba124db1, State: Running, Role: LEADER
I20260812 08:00:33.495893 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:33.496022 26057 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.496304 26302 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 5.55, Last appended by leader: 55, Current term: 6, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.497684 26057 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 6.
I20260812 08:00:33.500245 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 26470
I20260812 08:00:33.500154 24051 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 reported cstate change: term changed from 5 to 6, leader changed from e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132) to dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133). New cstate: current_term: 6 leader_uuid: "dbc6ed4943d5493caaf8b5feba124db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:33.515323 26626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:35833 every 8 connection(s)
W20260812 08:00:33.516837 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.519726 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.520404 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.522027 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.526081 24478 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.526917 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.527455 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.527875 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.530460 26631 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:33.530835 26631 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:33.531663 26631 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:33.532451 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.533972 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.534530 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.536286 24054 ts_manager.cc:194] Re-registered known tserver with Master: e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.537238 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:54463
W20260812 08:00:33.540279 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.541224 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.542845 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.543102 25756 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.543674 25754 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.544436 25759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.544896 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.544476 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
W20260812 08:00:33.545590 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.546546 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.546763 24767 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 6 election: Tablet error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:33.547592 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.548157 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.551998 26631 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:33.555437 25339 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 71ms of latency in SegmentAllocator::Sync()
W20260812 08:00:33.555459 26628 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:33.555544 26628 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 08:00:33.555593 26302 fault_injection.cc:59] injecting fault for kudu (pid 24751): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:33.568773 25211 connection.cc:573] client connection to 127.22.208.133:35369 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:33.569033 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:33.571568 26499 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=54 ignored=50} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 08:00:33.572201 26499 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap complete.
W20260812 08:00:33.572420 26499 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 68ms of latency in SegmentAllocator::Sync()
W20260812 08:00:33.572494 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.573918 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.574035 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.574285 23362 ts_itest-base.cc:575] TS 4 appears to have crashed. Restarting.
W20260812 08:00:33.574784 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.574875 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.133:35369
--local_ip_for_outbound_sockets=127.22.208.133
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=38847
--webserver_interface=127.22.208.133
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:33.576727 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.580991 26005 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.581277 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.582651 26005 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.582746 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.582801 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.583751 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.583918 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.585875 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.585891 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.589119 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.589362 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.590161 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.590680 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.608301 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.608301 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.609160 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.610240 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.610569 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.611843 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.618290 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.620428 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.620481 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.628083 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.631453 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.632174 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.641301 26499 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent bootstrapping tablet: real 0.179s	user 0.040s	sys 0.007s
W20260812 08:00:33.643122 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.643487 26499 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.644148 26499 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Initialized, Role: FOLLOWER
I20260812 08:00:33.645128 26499 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 5.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:33.645619 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.646148 26499 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
W20260812 08:00:33.647047 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.648311 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
W20260812 08:00:33.649034 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.649258 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20260812 08:00:33.649871 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.658465 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.664300 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.664367 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.665752 24325 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.665891 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.665901 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.666301 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.667325 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.667755 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.669150 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.673256 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.678856 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.678948 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.680117 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.680194 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.681816 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.682446 26352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.682821 26348 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.683274 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.683704 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.690853 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.692553 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.692824 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.694321 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.696827 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.698624 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.698959 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.700788 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.701817 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.702837 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.707355 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.709262 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.711100 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.713682 26505 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.715914 26505 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.719357 26509 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.719492 26505 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.719496 26504 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.719983 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.720556 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.723701 26004 tablet_service.cc:3080] error setting up scanner 2eb61c5919274f2687d14caa31f04e71 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.902s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "70ba6b37d84e4c5492a90938c729ec89"
W20260812 08:00:33.725754 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.731050 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.732023 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.733703 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.737608 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.739059 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.740407 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.741868 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.742345 26639 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:33.742631 26639 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:33.742702 26639 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:33.742746 26639 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:33.742825 26639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:33.742871 26639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:33.742971 26639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:33.747890 26639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:33.748092 26639 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:33.748163 26639 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:33.748270 26639 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.133
W20260812 08:00:33.753332 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.754506 26639 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.133:35369
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.22.208.133
--webserver_port=38847
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.26639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.133

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
W20260812 08:00:33.754659 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.756114 26639 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:33.757584 26639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:33.758174 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.758232 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.759428 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
E20260812 08:00:33.761600 26639 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:33.762975 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.766779 26645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:33.766820 26646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:33.767086 26648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:33.767812 26639 server_base.cc:1061] running on GCE node
I20260812 08:00:33.768381 26639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:33.769172 26639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:33.769701 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.769953 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.770709 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.773754 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.776947 26639 hybrid_clock.cc:648] HybridClock initialized: now 1786521633776057 us; error 919 us; skew 500 ppm
W20260812 08:00:33.777017 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.779737 26639 webserver.cc:533] Webserver started at http://127.22.208.133:38847/ using document root <none> and password file <none>
I20260812 08:00:33.780462 26639 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:33.780548 26639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:33.785704 26639 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.004s
W20260812 08:00:33.789407 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.791371 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.792670 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.793483 26660 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 08:00:33.793799 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.794983 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.795058 26639 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.002s	sys 0.000s
I20260812 08:00:33.795181 26639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "dbc6ed4943d5493caaf8b5feba124db1"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:33.796226 26639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 6
Total live bytes: 6310
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:33.797853 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.797875 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.799306 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.804522 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.808894 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.811041 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.811149 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.811602 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.815446 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.818298 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.819438 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.821782 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.821815 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.826157 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.832047 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.833518 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.833873 26639 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
W20260812 08:00:33.834772 26351 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.834775 26639 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20260812 08:00:33.834923 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.835005 26639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:33.835924 26639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 08:00:33.835963 26350 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52526: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.837425 26668 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:33.841414 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.841414 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.843145 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:33.844887 26639 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:33.845014 26639 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.000s	sys 0.001s
I20260812 08:00:33.845106 26639 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 08:00:33.845245 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:33.847477 26639 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:33.847590 26639 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:33.847966 26668 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap starting.
W20260812 08:00:33.851717 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.855312 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.856570 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.858848 26003 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:33.860738 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:33.862012 26639 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.133:35369
W20260812 08:00:33.862891 26004 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50098: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:33.863940 26639 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
W20260812 08:00:33.864128 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.864933 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.869618 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.870342 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.870836 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 26639
W20260812 08:00:33.877954 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.878772 26669 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:33.878949 26669 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:33.880976 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.883235 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.883389 26669 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:33.883975 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.885879 26800 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:33.886206 26800 heartbeater.cc:461] Registering TS with master...
W20260812 08:00:33.886165 26004 tablet_service.cc:3080] error setting up scanner e138df1ecf4848b9b6aecffb94534fa6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.065s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
I20260812 08:00:33.887036 26800 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:33.888763 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.888886 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.889945 24058 ts_manager.cc:194] Re-registered known tserver with Master: dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369)
W20260812 08:00:33.890527 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.891044 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.133:44705
I20260812 08:00:33.900425 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.900645 26057 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.900887 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:33.901134 26564 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 5: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:33.901487 26292 leader_election.cc:400] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Vote denied by peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.901551 26292 leader_election.cc:403] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260812 08:00:33.902395 26669 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 4 FOLLOWER]: Advancing to term 6
I20260812 08:00:33.902670 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.902859 24376 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.903230 26291 leader_election.cc:400] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.904074 26800 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:33.904696 26669 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.905304 26799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.133:35369 every 8 connection(s)
I20260812 08:00:33.910805 26816 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:33.910969 26816 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.912842 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.913031 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 6.
I20260812 08:00:33.913600 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.913771 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 6.
I20260812 08:00:33.915608 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.915776 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 6.
I20260812 08:00:33.914616 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.916208 25974 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: 
I20260812 08:00:33.916908 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.917080 24529 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
I20260812 08:00:33.917469 26292 leader_election.cc:400] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 6.
W20260812 08:00:33.917681 26292 leader_election.cc:343] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:33.917471 26398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:33.918447 26398 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 6.
I20260812 08:00:33.920647 26640 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:33.920879 26640 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.922493 26816 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.922798 26816 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260812 08:00:33.922932 26816 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:33.923046 26816 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:33.923301 26640 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:33.932143 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:33.932372 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 in current term 6: Already voted for candidate dbc6ed4943d5493caaf8b5feba124db1 in this term.
I20260812 08:00:33.932348 26816 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.936251 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:33.937266 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:33.937326 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:33.937438 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:33.937474 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:33.938761 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:33.938918 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:33.939138 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 7.
I20260812 08:00:33.940196 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 7.
W20260812 08:00:33.941125 25975 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:33.941241 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.941417 25787 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 7.
I20260812 08:00:33.941437 26398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:33.941629 26398 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:33.941818 26292 leader_election.cc:400] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for earlier term 5. Current term is 7.
I20260812 08:00:33.942284 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
W20260812 08:00:33.942821 25975 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 election: Tablet error from VoteRequest() call to peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:33.942910 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 7.
I20260812 08:00:33.942996 26398 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 7.
I20260812 08:00:33.943428 25974 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:33.943763 26819 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 FOLLOWER]: Leader election won for term 7
I20260812 08:00:33.944119 26819 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 LEADER]: Becoming Leader. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Running, Role: LEADER
I20260812 08:00:33.946396 26819 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 4.54, Last appended by leader: 54, Current term: 7, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:33.947597 26816 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 7 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:33.948664 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
W20260812 08:00:33.949303 26496 leader_election.cc:343] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Tablet error from VoteRequest() call to peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:33.951344 24058 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b reported cstate change: term changed from 6 to 7, leader changed from dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133) to b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134). New cstate: current_term: 7 leader_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:33.951897 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:33.952062 24529 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.953235 26496 leader_election.cc:400] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.953305 26496 leader_election.cc:403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260812 08:00:33.953593 26640 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 5 FOLLOWER]: Advancing to term 7
W20260812 08:00:33.953984 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.953989 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.954833 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:33.955714 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:33.956063 26554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 7 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
W20260812 08:00:33.956414 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.956802 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.958770 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.959959 26398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:33.960201 26398 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.960286 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:33.960465 25787 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.960924 26640 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.960924 26496 leader_election.cc:400] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.961318 26564 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b for term 7 because replica has last-logged OpId of term: 5 index: 55, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 54.
W20260812 08:00:33.962116 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:33.963327 26492 leader_election.cc:400] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 for earlier term 6. Current term is 7.
I20260812 08:00:33.969120 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 6 candidate_status { last_received { term: 5 index: 55 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:33.973768 26668 log.cc:826] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Log is configured to *not* fsync() on all Append() calls
W20260812 08:00:33.985488 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.985517 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 32ms of latency in SegmentAllocator::Sync()
W20260812 08:00:33.985579 26829 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:33.985628 26829 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:33.986949 26564 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 5 index: 55. Preceding OpId from leader: term: 7 index: 56. (index mismatch)
I20260812 08:00:33.987712 26829 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260812 08:00:33.987864 26829 consensus_queue.cc:1243] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Peer e3a0fdc906c74926946b47c61bd5cea5 log is divergent from this leader: its last log entry 5.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 54
W20260812 08:00:33.988433 25975 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b -> Peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Couldn't send request to peer dbc6ed4943d5493caaf8b5feba124db1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:33.988883 26564 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Aborting all ops after (but not including) 54
I20260812 08:00:33.988960 26564 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Aborting uncommitted NO_OP operation due to leader change: 5.55
I20260812 08:00:33.989037 26564 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:33.989444 26398 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 7 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 4 index: 54. Preceding OpId from leader: term: 7 index: 56. (index mismatch)
I20260812 08:00:33.990546 26829 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
W20260812 08:00:33.992975 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:33.996156 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:33.996243 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:34.000524 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:34.001376 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:34.003288 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:34.005822 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 4 index: 54. Preceding OpId from leader: term: 7 index: 56. (index mismatch)
I20260812 08:00:34.007015 26819 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260812 08:00:34.007476 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 4 index: 54. Preceding OpId from leader: term: 7 index: 56. (index mismatch)
I20260812 08:00:34.007611 25787 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 4 index: 54. Preceding OpId from leader: term: 7 index: 56. (index mismatch)
I20260812 08:00:34.008276 26816 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260812 08:00:34.008411 26819 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
W20260812 08:00:34.012006 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 4ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.013182 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 178ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.021335 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 124ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.021992 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 13ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.026636 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:34.033957 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 119ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.034022 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 101ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.036069 26668 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=54 ignored=50} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260812 08:00:34.036768 26668 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap complete.
W20260812 08:00:34.036984 26668 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 58ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.042265 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 147ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.042490 26841 mvcc.cc:204] Tried to move back new op lower bound from 7317592612801638400 to 7317592612653580288. Current Snapshot: MvccSnapshot[applied={T|T < 7317592612801638400}]
I20260812 08:00:34.043468 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.051188 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 44ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.052896 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.054199 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.095625 26668 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent bootstrapping tablet: real 0.248s	user 0.047s	sys 0.007s
W20260812 08:00:34.096149 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 24ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.098671 26668 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.101701 26668 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc6ed4943d5493caaf8b5feba124db1, State: Initialized, Role: FOLLOWER
I20260812 08:00:34.102305 26668 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 6.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.103402 26668 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260812 08:00:34.105494 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 526 bytes on disk
I20260812 08:00:34.106560 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.121987 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.069s	user 0.000s	sys 0.001s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":123,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.122498 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:34.125427 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 6ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.134116 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.011s	user 0.004s	sys 0.001s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":273,"cfile_cache_miss":2,"cfile_cache_miss_bytes":69,"cfile_init":1,"delete_count":21,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":307,"lbm_read_time_us":94,"lbm_reads_lt_1ms":6,"lbm_write_time_us":152,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":167,"threads_started":1,"update_count":0}
W20260812 08:00:34.142345 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 6ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.148844 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.151364 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.153842 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 62ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.155059 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 31ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.158908 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.007s	user 0.006s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":161,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.159511 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.160969 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 30ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.166340 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.168025 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 98ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.171726 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.173439 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:34.176263 26747 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Aborting all ops after (but not including) 54
I20260812 08:00:34.176375 26747 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Aborting uncommitted NO_OP operation due to leader change: 5.55
I20260812 08:00:34.176496 26747 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:34.176676 26747 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Aborting uncommitted NO_OP operation due to leader change: 6.56
I20260812 08:00:34.176724 26747 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:34.176779 26747 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Aborting uncommitted WRITE_OP operation due to leader change: 6.57
W20260812 08:00:34.182915 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 90ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.188223 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.016s	user 0.001s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":102,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.188731 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:34.189913 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 61ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.196440 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.037s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":121,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.197011 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.202692 26834 mvcc.cc:204] Tried to move back new op lower bound from 7317592612801638400 to 7317592612653580288. Current Snapshot: MvccSnapshot[applied={T|T < 7317592612801638400}]
I20260812 08:00:34.206158 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.207718 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 18ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.216964 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 23ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.218396 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.175s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":153,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.219800 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.226797 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 82ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.231218 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.235157 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 13ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.237433 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.026s	user 0.003s	sys 0.000s Metrics: {"cfile_cache_hit":4,"cfile_cache_hit_bytes":353,"delete_count":22,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_write_time_us":161,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":154,"threads_started":1,"update_count":0}
I20260812 08:00:34.238008 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.239102 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.008s	user 0.004s	sys 0.002s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":200,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:34.241128 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 40ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.245594 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 21ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.247179 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.251730 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 105ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.252650 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.198s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":160,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.253751 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.259249 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.021s	user 0.002s	sys 0.000s Metrics: {"bytes_written":231,"delete_count":8,"lbm_write_time_us":136,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.263634 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.266661 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.267876 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.101s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.268478 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.275823 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 45ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.276707 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 24ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.281937 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 123ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.283598 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.064s	user 0.002s	sys 0.000s Metrics: {"bytes_written":214,"delete_count":6,"lbm_write_time_us":127,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.284633 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.078s	user 0.003s	sys 0.003s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":176,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.285416 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.285802 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.287104 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 2ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.291311 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 80ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.296106 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.032s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4692,"cfile_init":1,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":18210,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":2}
I20260812 08:00:34.297499 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.309657 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.309707 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 3ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.310348 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.040s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":149,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.311005 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.311015 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.101s	user 0.002s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":150,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.311811 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.316943 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.019s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4666,"cfile_init":1,"dirs.queue_time_us":6520,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1,"thread_start_us":135,"threads_started":1}
I20260812 08:00:34.317787 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.320468 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 27ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.321269 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 94ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.324563 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 113ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.346782 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.035s	user 0.002s	sys 0.002s Metrics: {"bytes_written":204,"delete_count":5,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.347463 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.350836 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 64ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.355760 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.040s	user 0.003s	sys 0.000s Metrics: {"bytes_written":204,"delete_count":5,"lbm_write_time_us":1774,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":3,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.356352 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.357255 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 55ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.361644 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 60ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.372201 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 90ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.372936 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.083s	user 0.000s	sys 0.001s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":201,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.373543 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.405967 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 10ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.413635 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 147ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.414093 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.160s	user 0.002s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":114,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.415014 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.415352 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 27ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.417801 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.131s	user 0.003s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":123,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.417801 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.170s	user 0.000s	sys 0.002s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.418437 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.418462 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.421729 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.001s	sys 0.001s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":133,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.423614 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.067s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4892,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2984,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":9}
I20260812 08:00:34.424383 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.428845 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.429263 26830 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 31ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.430019 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 90ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.439734 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.121s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4748,"cfile_init":1,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":3278,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5}
W20260812 08:00:34.440379 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.440568 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.201351
I20260812 08:00:34.444747 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.097s	user 0.007s	sys 0.005s Metrics: {"bytes_written":4711,"cfile_init":1,"compiler_manager_pool.queue_time_us":2189,"dirs.queue_time_us":2002,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":2519,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10017,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":3,"thread_start_us":328,"threads_started":2}
I20260812 08:00:34.446066 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 185 bytes on disk
I20260812 08:00:34.446679 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.447351 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.456374 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.458024 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.462627 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.043s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":218,"dirs.run_cpu_time_us":295,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":155,"threads_started":1}
W20260812 08:00:34.463230 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 115ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.463693 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 254 bytes on disk
I20260812 08:00:34.469620 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.040s	user 0.009s	sys 0.001s Metrics: {"bytes_written":4908,"cfile_init":1,"compiler_manager_pool.queue_time_us":320,"dirs.queue_time_us":3206,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":17305,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":273,"threads_started":2}
I20260812 08:00:34.470813 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.966320
I20260812 08:00:34.471664 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.472373 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.966364
I20260812 08:00:34.496071 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.048s	user 0.000s	sys 0.005s Metrics: {"bytes_written":4781,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1161,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1316,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":7}
I20260812 08:00:34.496819 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.499714
W20260812 08:00:34.513170 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 21ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.522102 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.097s	user 0.004s	sys 0.000s Metrics: {"bytes_written":4665,"cfile_init":1,"dirs.queue_time_us":96,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1891,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1}
I20260812 08:00:34.522954 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.960393
I20260812 08:00:34.526302 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.053s	user 0.008s	sys 0.003s Metrics: {"bytes_written":5661,"cfile_cache_hit":22,"cfile_cache_hit_bytes":2014,"cfile_cache_miss":30,"cfile_cache_miss_bytes":3290,"cfile_init":7,"delta_iterators_relevant":12,"dirs.queue_time_us":31533,"dirs.run_cpu_time_us":400,"dirs.run_wall_time_us":1926,"drs_written":1,"lbm_read_time_us":774,"lbm_reads_lt_1ms":58,"lbm_write_time_us":4457,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":25,"mutex_wait_us":20179,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":40,"thread_start_us":222,"threads_started":3}
I20260812 08:00:34.527027 26145 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.527456 25982 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.548684 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.026s	user 0.016s	sys 0.000s Metrics: {"bytes_written":5661,"cfile_cache_hit":35,"cfile_cache_hit_bytes":3298,"cfile_cache_miss":35,"cfile_cache_miss_bytes":2680,"cfile_init":9,"delta_iterators_relevant":14,"drs_written":1,"lbm_read_time_us":1228,"lbm_reads_lt_1ms":71,"lbm_write_time_us":828,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":40,"thread_start_us":397,"threads_started":6}
I20260812 08:00:34.549230 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.549741 24306 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.549935 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.076s	user 0.019s	sys 0.000s Metrics: {"bytes_written":5661,"cfile_cache_hit":22,"cfile_cache_hit_bytes":2036,"cfile_cache_miss":30,"cfile_cache_miss_bytes":3286,"cfile_init":8,"delta_iterators_relevant":12,"dirs.queue_time_us":7468,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1035,"drs_written":1,"lbm_read_time_us":928,"lbm_reads_lt_1ms":62,"lbm_write_time_us":903,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":40,"thread_start_us":566,"threads_started":5}
I20260812 08:00:34.550756 26632 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.550426 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.110s	user 0.019s	sys 0.000s Metrics: {"bytes_written":5633,"cfile_cache_hit":27,"cfile_cache_hit_bytes":2158,"cfile_cache_miss":53,"cfile_cache_miss_bytes":3948,"cfile_init":13,"delta_iterators_relevant":12,"dirs.queue_time_us":3758,"dirs.run_cpu_time_us":613,"dirs.run_wall_time_us":5730,"drs_written":1,"lbm_read_time_us":1753,"lbm_reads_lt_1ms":105,"lbm_write_time_us":756,"lbm_writes_lt_1ms":26,"num_input_rowsets":4,"peak_mem_usage":131072,"rows_written":38,"thread_start_us":7506,"threads_started":2}
I20260812 08:00:34.551901 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 632 bytes on disk
I20260812 08:00:34.553584 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.554293 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.557868 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 58ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.561352 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 65ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.562691 26497 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4}
W20260812 08:00:34.574520 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 41ms of latency in SegmentAllocator::Sync()
W20260812 08:00:34.578617 26665 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260812 08:00:34.579890 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.206s	user 0.001s	sys 0.000s Metrics: {"bytes_written":231,"delete_count":8,"lbm_write_time_us":148,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.580399 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:34.596943 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 44ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.617684 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.063s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4694,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1862,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2}
I20260812 08:00:34.618414 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.351783
W20260812 08:00:34.627045 25724 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260812 08:00:34.628393 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.204s	user 0.002s	sys 0.000s Metrics: {"bytes_written":204,"delete_count":5,"lbm_write_time_us":134,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:34.628700 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.132s	user 0.015s	sys 0.007s Metrics: {"bytes_written":5661,"cfile_cache_hit":31,"cfile_cache_hit_bytes":2270,"cfile_cache_miss":43,"cfile_cache_miss_bytes":3632,"cfile_init":10,"delta_iterators_relevant":16,"dirs.queue_time_us":2604,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1367,"drs_written":1,"lbm_read_time_us":5524,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":82,"lbm_write_time_us":11910,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":24,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":40,"spinlock_wait_cycles":2428288,"thread_start_us":481,"threads_started":5}
I20260812 08:00:34.629077 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:34.629298 26460 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.630508 26303 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20260812 08:00:34.638514 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 67ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.643579 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.063s	user 0.005s	sys 0.008s Metrics: {"bytes_written":4908,"cfile_init":1,"compiler_manager_pool.queue_time_us":195,"dirs.queue_time_us":40819,"dirs.run_cpu_time_us":337,"dirs.run_wall_time_us":7686,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":226,"threads_started":2}
I20260812 08:00:34.644909 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.966388
I20260812 08:00:34.667038 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.043s	user 0.011s	sys 0.003s Metrics: {"bytes_written":5661,"cfile_cache_hit":11,"cfile_cache_hit_bytes":990,"cfile_cache_miss":29,"cfile_cache_miss_bytes":4134,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":19032,"dirs.run_cpu_time_us":805,"dirs.run_wall_time_us":8069,"drs_written":1,"lbm_read_time_us":1077,"lbm_reads_lt_1ms":57,"lbm_write_time_us":696,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":40,"thread_start_us":90,"threads_started":1}
I20260812 08:00:34.669384 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.670210 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.691385 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.045s	user 0.020s	sys 0.001s Metrics: {"bytes_written":5661,"cfile_cache_hit":22,"cfile_cache_hit_bytes":2014,"cfile_cache_miss":30,"cfile_cache_miss_bytes":3280,"cfile_init":8,"delta_iterators_relevant":12,"dirs.queue_time_us":1384,"dirs.run_cpu_time_us":375,"dirs.run_wall_time_us":2174,"drs_written":1,"lbm_read_time_us":1371,"lbm_reads_lt_1ms":62,"lbm_write_time_us":2387,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":25,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":40,"thread_start_us":5077,"threads_started":3}
I20260812 08:00:34.692204 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.693766 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.707389 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.078s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"dirs.queue_time_us":239,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":3935,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":10,"thread_start_us":96,"threads_started":1}
I20260812 08:00:34.708092 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.966359
E20260812 08:00:34.714845 26819 fault_injection.cc:59] injecting fault for kudu (pid 25918): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:34.724870 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 44ms of latency in SegmentAllocator::Sync()
I20260812 08:00:34.732445 23362 ts_itest-base.cc:575] TS 5 appears to have crashed. Restarting.
I20260812 08:00:34.733006 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:44711
--local_ip_for_outbound_sockets=127.22.208.134
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37749
--webserver_interface=127.22.208.134
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:34.772203 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.064s	user 0.011s	sys 0.002s Metrics: {"bytes_written":5661,"cfile_cache_hit":22,"cfile_cache_hit_bytes":2014,"cfile_cache_miss":30,"cfile_cache_miss_bytes":3292,"cfile_init":8,"delta_iterators_relevant":12,"drs_written":1,"lbm_read_time_us":861,"lbm_reads_lt_1ms":62,"lbm_write_time_us":464,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":40,"thread_start_us":328,"threads_started":5}
I20260812 08:00:34.773327 25858 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:34.774279 25724 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20260812 08:00:34.907608 26939 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:34.907860 26939 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.909198 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:34.909631 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:34.910384 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:34.917001 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:34.917652 26492 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:34.918517 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:34.918736 26939 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:34.918843 26939 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 08:00:34.919098 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:34.919224 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.920382 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:34.920718 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:34.922765 26496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:34.923310 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:34.930825 24305 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:34.931571 26496 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:34.933437 26933 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:34.933858 26933 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:34.933939 26933 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:34.934005 26933 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:34.934122 26933 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:34.934191 26933 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:34.934311 26933 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 08:00:34.935889 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:34.937247 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:34.937961 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
W20260812 08:00:34.939414 26933 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:34.939567 26933 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:34.939616 26933 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:34.939718 26933 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
I20260812 08:00:34.944336 26940 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 08:00:34.944602 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:34.944818 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 7.
I20260812 08:00:34.946797 26933 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:44711
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=37749
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.26933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:34.948454 26933 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:34.950071 26933 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:34.954109 26933 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:34.959127 26964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:34.959082 26961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:34.959054 26960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:34.960843 26933 server_base.cc:1061] running on GCE node
I20260812 08:00:34.961470 26933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:34.962374 26933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:34.963590 26933 hybrid_clock.cc:648] HybridClock initialized: now 1786521634963518 us; error 94 us; skew 500 ppm
I20260812 08:00:34.966061 26933 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:34.966843 26971 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:34.966971 26933 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:34.967078 26933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:34.967235 26971 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.968364 26971 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:34.969306 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:34.971467 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:34.971673 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 7.
I20260812 08:00:34.974140 26933 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.003s
I20260812 08:00:34.979777 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:34.979911 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:34.979970 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 7.
I20260812 08:00:34.981984 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:34.985133 26982 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:34.985353 26982 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:34.985656 25723 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:34.985757 25723 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:34.985822 26986 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:34.985945 26971 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 08:00:34.986476 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:34.986658 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 7.
I20260812 08:00:34.986958 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:34.987095 26933 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.002s	sys 0.000s
I20260812 08:00:34.987115 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 7.
I20260812 08:00:34.987713 26982 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:34.987223 26933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:34.987830 26933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 6
Total live bytes: 6843
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:34.992890 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:34.993069 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 7.
I20260812 08:00:34.993492 24450 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 
I20260812 08:00:34.993773 26982 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260812 08:00:34.993880 26982 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:34.993978 26982 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:34.995332 26982 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:34.996418 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:34.996539 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:34.996711 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:34.997499 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:34.997646 26564 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:34.998217 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 8.
I20260812 08:00:34.998507 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:34.998692 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:34.998878 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 8.
I20260812 08:00:34.999416 26982 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 election: Requested vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:34.999886 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 8.
I20260812 08:00:35.000633 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
W20260812 08:00:35.000941 24453 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.001295 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:35.003767 26982 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 08:00:35.004083 26982 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 LEADER]: Becoming Leader. State: Replica: 9dede2273d3a4266a1bf7a5b368080f8, State: Running, Role: LEADER
I20260812 08:00:35.004568 26982 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 7.75, Last appended by leader: 75, Current term: 8, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.006898 26999 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:35.007066 26999 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:35.008250 24453 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.008478 24050 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 reported cstate change: term changed from 7 to 8, leader changed from b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134) to 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131). New cstate: current_term: 8 leader_uuid: "9dede2273d3a4266a1bf7a5b368080f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:35.012380 26999 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.012519 26398 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:35.012676 26398 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:35.013919 26398 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 8.
I20260812 08:00:35.014303 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.014454 26402 raft_consensus.cc:2376] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 8. Re-sending same reply.
W20260812 08:00:35.018195 26664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.019148 26402 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.019346 26402 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 8: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:35.019518 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.019704 25787 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 8: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:35.021374 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.023693 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:35.023898 26564 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 8: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:35.024380 26661 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:35.024843 26999 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 7 FOLLOWER]: Advancing to term 8
W20260812 08:00:35.025213 26664 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.025492 26933 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:35.025576 27000 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:35.025698 27000 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.026393 26933 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:35.026631 26933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:35.026822 26999 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260812 08:00:35.027287 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.027379 26933 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:35.027411 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.027463 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 8.
I20260812 08:00:35.027464 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.027671 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 8.
I20260812 08:00:35.029237 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.029423 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 8.
I20260812 08:00:35.029769 26292 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 9dede2273d3a4266a1bf7a5b368080f8
I20260812 08:00:35.030009 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 8 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.030256 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 8: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:35.030755 27000 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:35.030915 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.030977 27000 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260812 08:00:35.031061 27000 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:35.031121 27000 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.031464 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:35.031651 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 8.
I20260812 08:00:35.032477 27019 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:35.033146 26292 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.035758 27000 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.037235 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:35.037395 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:35.037521 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.037833 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:35.037984 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.038713 25787 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 9.
I20260812 08:00:35.038731 26933 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:35.038798 26933 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
I20260812 08:00:35.038879 26933 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:35.039258 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 9.
I20260812 08:00:35.040316 27000 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.040556 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:35.040684 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.040827 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 9 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:35.040971 26564 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.041083 26933 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:35.041143 26933 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 08:00:35.041847 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 9.
I20260812 08:00:35.042167 26564 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 9.
I20260812 08:00:35.042750 26292 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 9dede2273d3a4266a1bf7a5b368080f8
W20260812 08:00:35.043402 24328 tablet_service.cc:3080] error setting up scanner 5e876d6b8533495a860f5c79105a3d3c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.432s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592607664189440 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "19f7ff22fdb7433eaa325ec3ee25d231"
W20260812 08:00:35.043644 26292 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:35.044586 27000 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Leader election won for term 9
I20260812 08:00:35.044735 27000 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 LEADER]: Becoming Leader. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Running, Role: LEADER
I20260812 08:00:35.045117 27000 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 7.75, Last appended by leader: 75, Current term: 9, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.047224 27019 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
W20260812 08:00:35.048017 26837 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 110ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.049904 24055 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 reported cstate change: term changed from 8 to 9, leader changed from 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131) to f29b76cef775415dbd2c0997fadb3904 (127.22.208.129). New cstate: current_term: 9 leader_uuid: "f29b76cef775415dbd2c0997fadb3904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:35.058261 26933 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:35.058290 27151 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
I20260812 08:00:35.059772 26933 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:35.063083 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 26933
I20260812 08:00:35.070564 27019 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:35.071676 27152 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:35.071975 27152 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:35.072585 27152 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:35.074663 24054 ts_manager.cc:194] Re-registered known tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:35.075640 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:40249
W20260812 08:00:35.081390 26997 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:35.081493 26997 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:35.082286 26402 raft_consensus.cc:1240] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 LEADER]: Rejecting Update request from peer 9dede2273d3a4266a1bf7a5b368080f8 for earlier term 8. Current term is 9. Ops: []
I20260812 08:00:35.083051 26997 consensus_queue.cc:1059] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: INVALID_TERM, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.083294 26997 consensus_queue.cc:1243] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [LEADER]: Peer f29b76cef775415dbd2c0997fadb3904 log is divergent from this leader: its last log entry 9.76 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 75
I20260812 08:00:35.083437 26982 raft_consensus.cc:3055] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 08:00:35.083545 26982 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9dede2273d3a4266a1bf7a5b368080f8, State: Running, Role: LEADER
I20260812 08:00:35.083729 26982 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 8.76, Last appended by leader: 76, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.083990 26982 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.088264 27152 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:35.111430 27019 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:35.112010 27019 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap complete.
W20260812 08:00:35.112190 27019 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 19ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.131647 27019 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.085s	user 0.048s	sys 0.009s
I20260812 08:00:35.134150 27019 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.134747 27019 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
W20260812 08:00:35.134886 27000 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:35.134944 27000 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:35.135757 27019 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 7.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.135864 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 8 index: 76. Preceding OpId from leader: term: 9 index: 76. (term mismatch)
I20260812 08:00:35.135954 24529 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Aborting all ops after (but not including) 75
I20260812 08:00:35.136023 24529 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Aborting uncommitted NO_OP operation due to leader change: 8.76
I20260812 08:00:35.136065 24529 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:35.136711 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.136929 27019 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:00:35.142040 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 9 index: 76. (index mismatch)
I20260812 08:00:35.143034 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.143105 27153 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
W20260812 08:00:35.144342 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 119ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.144197 27017 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20260812 08:00:35.151841 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 9 index: 76. (index mismatch)
I20260812 08:00:35.152549 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.153118 25787 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 9 index: 76. (index mismatch)
I20260812 08:00:35.153638 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:35.154994 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 111ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.157157 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 77ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.158313 25323 scanner-internal.cc:458] Time spent opening tablet: real 2.276s	user 0.004s	sys 0.000s
I20260812 08:00:35.170068 27078 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 7 FOLLOWER]: Advancing to term 9
I20260812 08:00:35.171030 26564 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 9 index: 76. (index mismatch)
I20260812 08:00:35.171546 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.172956 27078 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 9 index: 76. (index mismatch)
I20260812 08:00:35.174362 27000 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
E20260812 08:00:35.174628 27000 fault_injection.cc:59] injecting fault for kudu (pid 26248): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:35.196620 23362 ts_itest-base.cc:575] TS 0 appears to have crashed. Restarting.
I20260812 08:00:35.197371 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:40873
--local_ip_for_outbound_sockets=127.22.208.129
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37489
--webserver_interface=127.22.208.129
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:35.234764 24302 connection.cc:466] server connection from 127.22.208.129:40989 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.129:40989 (request call id 4) could send its response
W20260812 08:00:35.252846 27033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.252846 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.257454 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.258893 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.261859 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.264104 26663 connection.cc:466] server connection from 127.22.208.129:56073 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.129:56073 (request call id 4) could send its response
W20260812 08:00:35.265152 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.267248 25723 connection.cc:466] server connection from 127.22.208.129:49137 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.129:49137 (request call id 4) could send its response
W20260812 08:00:35.269083 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.273190 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.276647 25208 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
W20260812 08:00:35.280025 26674 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.283535 26674 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.283779 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.290313 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.290833 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.291671 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.298055 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.306496 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.308650 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.309764 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.311782 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.311812 27034 tablet_service.cc:3080] error setting up scanner de8ff6f00d6f458ea2ba15f93a21ba72 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.179s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592617602342912 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "cfb00f1c05764247815c94a767d5c597"
W20260812 08:00:35.313190 24480 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.313400 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.313545 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.314989 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.315102 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.317762 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.318356 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.320041 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.320679 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.321143 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.321324 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.324082 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.325312 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.327708 27033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.328394 27036 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.328436 27033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.329216 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.329725 27036 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.330046 27033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.330245 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.332507 27034 tablet_service.cc:3080] error setting up scanner 476cbe9d388547a7b24a8c3a33dc5e52 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.200s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592617602342912 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "cfb00f1c05764247815c94a767d5c597"
W20260812 08:00:35.334712 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.337488 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.338670 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.339119 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.339417 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.343521 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.345268 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.345340 26674 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.346654 26674 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.346782 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.347257 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.347461 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.346694 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.350737 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.353281 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.354312 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.356420 27160 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:35.356781 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.356998 27160 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:35.357053 27160 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:35.357079 27160 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:35.357251 27160 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:35.357314 27160 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:35.357419 27160 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:35.358420 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.360447 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.361665 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.361665 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.363066 27160 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:35.363229 27160 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:35.363272 27160 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:35.363382 27160 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
W20260812 08:00:35.363896 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.365186 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.365252 27034 tablet_service.cc:3080] error setting up scanner 06881f6f9cc1478b9990710b700641c7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.233s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592617602342912 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "cfb00f1c05764247815c94a767d5c597"
W20260812 08:00:35.367213 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:35.370033 27160 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:40873
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=37489
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.27160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:35.371807 27160 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260812 08:00:35.371809 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.373950 27160 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:35.373907 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.374444 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
E20260812 08:00:35.377738 27160 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:35.381138 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.383234 27166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:35.383244 27171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:35.384374 27160 server_base.cc:1061] running on GCE node
W20260812 08:00:35.384438 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.385903 27169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:35.386422 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.386458 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:35.387243 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:35.388298 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.388475 27160 hybrid_clock.cc:648] HybridClock initialized: now 1786521635388438 us; error 48 us; skew 500 ppm
W20260812 08:00:35.389509 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.390954 27160 webserver.cc:533] Webserver started at http://127.22.208.129:37489/ using document root <none> and password file <none>
I20260812 08:00:35.391601 27160 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:35.391692 27160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:35.395323 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.396692 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.397670 27160 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.003s
W20260812 08:00:35.397814 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.398313 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.399689 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.404677 27182 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
W20260812 08:00:35.405638 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.405977 27160 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.002s	sys 0.000s
I20260812 08:00:35.406102 27160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:35.406670 27160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 6
Total live bytes: 6843
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:35.407328 27157 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:35.407506 27157 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.409397 27157 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:35.412460 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.413064 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.413741 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.416029 26992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:35.416586 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 10 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:35.419062 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.421752 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.421857 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 10 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
W20260812 08:00:35.422274 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.424830 26992 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
W20260812 08:00:35.426440 24326 tablet_service.cc:3080] error setting up scanner 662dabeb72d447c9a6534fdb6f4dd45b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.273s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592617602342912 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "cfb00f1c05764247815c94a767d5c597"
I20260812 08:00:35.427613 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 10 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.427989 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 10 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.428043 26995 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:35.428329 27157 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260812 08:00:35.429494 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.429608 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.430395 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.430645 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 10 candidate_status { last_received { term: 7 index: 75 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:35.431402 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.432719 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.437023 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.437889 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.437919 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:35.438258 26982 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.438375 26982 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.439193 26982 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.439327 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.439666 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.440270 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.441139 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:35.442006 24450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20260812 08:00:35.442296 24450 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.442489 26982 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260812 08:00:35.442946 27033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.443050 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
W20260812 08:00:35.443463 27036 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.443573 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.443993 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.446045 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:35.446798 27078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.447010 27078 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 9.
I20260812 08:00:35.447239 27160 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
W20260812 08:00:35.447713 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:35.448071 27160 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:35.448365 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:35.449074 27160 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 08:00:35.449937 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:35.450322 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.450678 27197 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:35.454617 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.454756 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.456274 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.456272 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.456434 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 9.
I20260812 08:00:35.456506 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.457612 27160 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:35.457751 27160 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
W20260812 08:00:35.457764 24302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20260812 08:00:35.457849 27160 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 08:00:35.458385 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.459100 24302 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:35.459538 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.459899 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:35.460307 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:35.460506 26940 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 08:00:35.460866 27160 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:35.460964 27160 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:00:35.461362 27197 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap starting.
W20260812 08:00:35.461726 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.462318 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.463213 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.464720 26506 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.465330 26507 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40746: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.466018 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.467907 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.473569 26941 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.473734 26941 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.474156 27160 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:40873
I20260812 08:00:35.474860 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.475097 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.475235 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 9.
I20260812 08:00:35.475283 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.475430 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.475569 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 9.
I20260812 08:00:35.476169 26941 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:35.476670 27160 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
W20260812 08:00:35.479460 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.480461 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.481036 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.482853 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.482985 27078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.483165 27078 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 9.
I20260812 08:00:35.483389 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 27160
W20260812 08:00:35.483726 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.485903 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.487912 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:35.488469 27078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:35.488554 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.488634 27078 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 9.
W20260812 08:00:35.488889 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.488994 26496 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:35.489229 26941 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260812 08:00:35.489298 26941 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.489351 26941 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.490744 26941 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.491493 26941 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:35.491711 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:35.491755 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:35.491885 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.492276 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:35.492436 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.492617 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:35.492996 27078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:35.493055 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 10.
I20260812 08:00:35.493141 27078 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.493749 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 10.
I20260812 08:00:35.494302 27078 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 10.
I20260812 08:00:35.494719 26496 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:35.494962 26941 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Leader election won for term 10
I20260812 08:00:35.495155 26941 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 LEADER]: Becoming Leader. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Running, Role: LEADER
I20260812 08:00:35.495529 26941 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 9.76, Last appended by leader: 76, Current term: 10, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.497748 27197 log.cc:826] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:35.499423 27329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:40873 every 8 connection(s)
I20260812 08:00:35.504329 24050 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 reported cstate change: term changed from 9 to 10, leader changed from f29b76cef775415dbd2c0997fadb3904 (127.22.208.129) to e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132). New cstate: current_term: 10 leader_uuid: "e3a0fdc906c74926946b47c61bd5cea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: HEALTHY } } }
I20260812 08:00:35.508728 27333 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:35.509032 27333 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:35.509791 27333 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:35.510077 24328 tablet_service.cc:3080] error setting up scanner f658bec22158434ea8f48c675e802126 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.356s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592617602342912 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "cfb00f1c05764247815c94a767d5c597"
I20260812 08:00:35.513381 24054 ts_manager.cc:194] Re-registered known tserver with Master: f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873)
I20260812 08:00:35.514393 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:55311
I20260812 08:00:35.527614 27333 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:35.531986 27275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.534332 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
W20260812 08:00:35.539453 26492 leader_election.cc:343] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:35.539813 26492 leader_election.cc:343] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 10 election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:35.548187 27197 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 08:00:35.548879 27197 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap complete.
W20260812 08:00:35.549116 27197 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 27ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.549193 26999 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.549355 26999 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.550338 26999 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.550823 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.551023 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 10: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:35.551111 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.551623 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:35.552030 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.552172 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 10: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:35.552877 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.553357 26664 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.553575 26999 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.554705 26999 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260812 08:00:35.556384 26971 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f29b76cef775415dbd2c0997fadb3904)
I20260812 08:00:35.556535 26971 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.557446 26971 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.557709 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.557919 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 10: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:35.558846 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.559561 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.559791 26564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:35.561571 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.561564 25761 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:35.562641 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.562808 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 10: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
W20260812 08:00:35.563654 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.563776 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.564144 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.564719 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.564918 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 10.
W20260812 08:00:35.566430 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.567584 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:35.568077 26661 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 10 pre-election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:35.570956 26836 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 87ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.571025 27336 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:35.571061 27336 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:35.571812 27078 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 7 index: 75. Preceding OpId from leader: term: 10 index: 78. (index mismatch)
I20260812 08:00:35.572010 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.572208 27088 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 10: Already voted for candidate e3a0fdc906c74926946b47c61bd5cea5 in this term.
I20260812 08:00:35.572697 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 10 index: 78. (index mismatch)
I20260812 08:00:35.572654 27336 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.573650 27336 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.576681 27197 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent bootstrapping tablet: real 0.116s	user 0.056s	sys 0.004s
W20260812 08:00:35.577481 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 101ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.578805 27197 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.579351 24529 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 10 index: 78. (index mismatch)
I20260812 08:00:35.579449 27197 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Initialized, Role: FOLLOWER
W20260812 08:00:35.579686 26492 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 -> Peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Couldn't send request to peer f29b76cef775415dbd2c0997fadb3904. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:35.579916 26941 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.579936 27197 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 9.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.581048 27197 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
W20260812 08:00:35.581881 26852 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 69ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.583163 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 10 index: 78. (index mismatch)
I20260812 08:00:35.583869 25787 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:35.584494 27336 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:35.585253 25787 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 10 index: 79. (index mismatch)
I20260812 08:00:35.585829 27336 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:35.586416 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 61ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.587491 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.588131 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 13ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.588162 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.588800 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.588891 24326 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:35.590324 27334 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
W20260812 08:00:35.590361 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.591317 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:35.591506 27195 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20260812 08:00:35.593510 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.593850 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 9.
W20260812 08:00:35.595562 27347 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 46ms of latency in SegmentAllocator::Sync()
I20260812 08:00:35.596634 27078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 10 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.597157 25723 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.597517 26971 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260812 08:00:35.605235 26846 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 55ms of latency in SegmentAllocator::Sync()
W20260812 08:00:35.607667 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.645789 27347 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 36ms of latency in SegmentAllocator::Sync()
E20260812 08:00:35.650221 27354 fault_injection.cc:59] injecting fault for kudu (pid 26470): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:35.675957 25722 connection.cc:573] server connection from 127.22.208.132:44131 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:35.676438 24453 connection.cc:573] server connection from 127.22.208.132:47933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:35.677732 24042 connection.cc:573] server connection from 127.22.208.132:54463 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:35.678516 23362 ts_itest-base.cc:575] TS 3 appears to have crashed. Restarting.
I20260812 08:00:35.679123 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:35833
--local_ip_for_outbound_sockets=127.22.208.132
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37633
--webserver_interface=127.22.208.132
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:35.681329 26661 connection.cc:466] server connection from 127.22.208.132:34229 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.132:34229 (request call id 5) could send its response
W20260812 08:00:35.682392 26994 connection.cc:466] server connection from 127.22.208.132:50365 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.132:50365 (request call id 4) could send its response
W20260812 08:00:35.692456 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.694199 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.698395 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.698395 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.701396 25757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58402: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:35.729367 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:35.836035 27356 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:35.836292 27356 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:35.836310 27356 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:35.836325 27356 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:35.836365 27356 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:35.836385 27356 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:35.836457 27356 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:35.841286 27356 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:35.841478 27356 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:35.841528 27356 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:35.841645 27356 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:00:35.848567 27356 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:35833
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=37633
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.27356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:35.850069 27356 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:35.851630 27356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:35.855741 27356 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:35.861274 27363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:35.861284 27361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:35.861285 27365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:35.862229 27356 server_base.cc:1061] running on GCE node
I20260812 08:00:35.862815 27356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:35.863726 27356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:35.864938 27356 hybrid_clock.cc:648] HybridClock initialized: now 1786521635864907 us; error 37 us; skew 500 ppm
I20260812 08:00:35.869405 27356 webserver.cc:533] Webserver started at http://127.22.208.132:37633/ using document root <none> and password file <none>
I20260812 08:00:35.870131 27356 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:35.870208 27356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:35.875622 27356 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:35.881564 26982 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.881705 26982 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.882640 26982 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.883038 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.883481 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.883728 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.884714 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
W20260812 08:00:35.885368 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:35.885452 24450 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.885699 26982 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260812 08:00:35.885869 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.885980 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.886729 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.887365 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 11 candidate_status { last_received { term: 10 index: 79 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.887441 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 11 candidate_status { last_received { term: 10 index: 79 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.887519 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 10.
I20260812 08:00:35.887984 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 11 candidate_status { last_received { term: 10 index: 79 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.888340 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 11 candidate_status { last_received { term: 10 index: 79 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:35.889554 24302 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:35.889631 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.889811 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.889899 26940 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260812 08:00:35.889997 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 9.
I20260812 08:00:35.890300 27382 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:35.891901 27356 fs_manager.cc:730] Time spent opening block manager: real 0.015s	user 0.004s	sys 0.000s
I20260812 08:00:35.892047 27356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:35.892659 27356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 6
Total live bytes: 6843
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:35.893987 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 11 candidate_status { last_received { term: 10 index: 79 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.894161 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 9.
I20260812 08:00:35.905265 26971 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.905413 26971 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.906114 26971 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.906399 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.906559 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 10.
I20260812 08:00:35.906703 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.907162 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.907491 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.907613 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.907622 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 9.
I20260812 08:00:35.907748 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 10.
I20260812 08:00:35.909837 25723 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, f29b76cef775415dbd2c0997fadb3904; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:35.910178 26971 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260812 08:00:35.910470 26971 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Starting leader election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.910557 26971 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.912539 26971 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.913537 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:35.913537 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:35.913537 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:35.913673 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.913688 24376 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.913692 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 9 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.915192 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 11.
I20260812 08:00:35.915468 24376 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 11.
I20260812 08:00:35.916415 26971 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:35.917219 25720 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:35.917335 25720 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:35.917352 27356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:35.917750 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:35.918242 27356 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:35.918197 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 11 candidate_status { last_received { term: 10 index: 81 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:35.916316 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 11.
I20260812 08:00:35.918526 27356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:35.918823 25720 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, f29b76cef775415dbd2c0997fadb3904; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:35.919068 26971 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Leader election won for term 11
I20260812 08:00:35.919411 26971 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 LEADER]: Becoming Leader. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Running, Role: LEADER
I20260812 08:00:35.919552 27356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:35.919900 26971 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 10.81, Last appended by leader: 81, Current term: 11, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.922648 27396 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:35.923983 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f reported cstate change: term changed from 10 to 11, leader changed from e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132) to 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135). New cstate: current_term: 11 leader_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:35.930073 27356 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:35.930328 27356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.000s
I20260812 08:00:35.930445 27356 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:35.932825 27356 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:35.932947 27356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:35.933334 27396 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap starting.
I20260812 08:00:35.945057 27157 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.945472 27157 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.946355 27157 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.947829 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:35.948205 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.948385 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.948441 27356 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:35833
I20260812 08:00:35.949138 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.949529 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.949690 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.950080 27356 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 08:00:35.951629 26995 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: b6d90cd6408d4dc3b47ebf57ca72e91b; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:35.952001 27157 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.952804 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 27356
I20260812 08:00:35.953179 27157 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260812 08:00:35.955490 27525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:35833 every 8 connection(s)
I20260812 08:00:35.955657 27396 log.cc:826] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:35.960013 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.960343 27274 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6d90cd6408d4dc3b47ebf57ca72e91b in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.970268 27526 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:35.971220 27526 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:35.972033 27526 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:35.973848 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 11 candidate_status { last_received { term: 10 index: 87 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:35.977200 26999 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:35.976953 24058 ts_manager.cc:194] Re-registered known tserver with Master: e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.977342 26999 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:35.978029 26999 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:35.978345 25787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:35.978483 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:34073
I20260812 08:00:35.978886 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:35.979058 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.979126 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:35.979344 27088 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 for term 11 because replica has last-logged OpId of term: 10 index: 87, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 78.
I20260812 08:00:35.979403 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:35.979561 24376 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.980090 26663 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:35.980463 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:35.980656 27274 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 11: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:35.980904 26999 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 10 FOLLOWER]: Advancing to term 11
I20260812 08:00:35.981999 26999 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260812 08:00:35.982362 26992 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:35.988129 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 11 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:35.988680 26661 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 11 pre-election: Tablet error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:35.994201 27526 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:36.000200 27396 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=1 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20260812 08:00:36.000406 26971 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:36.000469 26971 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:36.000862 27396 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap complete.
W20260812 08:00:36.001063 27396 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 89ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.004132 24376 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 10 index: 79. Preceding OpId from leader: term: 11 index: 82. (index mismatch)
I20260812 08:00:36.004662 26971 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:36.006754 25720 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f -> Peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Couldn't send request to peer e3a0fdc906c74926946b47c61bd5cea5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260812 08:00:36.007776 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 135ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.014549 27088 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 10 index: 87. Preceding OpId from leader: term: 11 index: 82. (term mismatch)
I20260812 08:00:36.014724 27088 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting all ops after (but not including) 81
I20260812 08:00:36.014810 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.82
I20260812 08:00:36.015424 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.83
I20260812 08:00:36.015769 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.84
I20260812 08:00:36.016070 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.85
I20260812 08:00:36.016341 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.86
I20260812 08:00:36.016583 27088 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 10.87
I20260812 08:00:36.017387 26971 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:36.018682 27347 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 66ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.019927 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 11 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 10 index: 78. Preceding OpId from leader: term: 11 index: 82. (index mismatch)
I20260812 08:00:36.020457 26971 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:36.022526 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 36ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.030308 27274 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 11 index: 82. (index mismatch)
I20260812 08:00:36.031011 26971 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 75, Time since last communication: 0.000s
E20260812 08:00:36.031227 26971 fault_injection.cc:59] injecting fault for kudu (pid 25646): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:36.036928 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.040462 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.041661 27034 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.041755 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.042968 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.045789 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:36.055209 23362 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 08:00:36.055634 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:33345
--local_ip_for_outbound_sockets=127.22.208.135
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37235
--webserver_interface=127.22.208.135
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:36.059142 26661 connection.cc:466] server connection from 127.22.208.135:49509 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:49509 (request call id 5) could send its response
W20260812 08:00:36.061826 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.065502 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.075292 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.085459 26995 connection.cc:466] server connection from 127.22.208.135:38749 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:38749 (request call id 4) could send its response
W20260812 08:00:36.087040 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.090674 27396 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent bootstrapping tablet: real 0.158s	user 0.058s	sys 0.007s
I20260812 08:00:36.092572 27396 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.094357 27396 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Initialized, Role: FOLLOWER
W20260812 08:00:36.094477 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.094988 27396 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 10.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:36.097656 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.097966 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.099489 27396 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
W20260812 08:00:36.103953 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.108614 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:36.109771 27391 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20260812 08:00:36.110603 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.113927 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.120745 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.128911 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.130383 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.137420 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.143430 24302 connection.cc:466] server connection from 127.22.208.135:42505 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:42505 (request call id 7) could send its response
W20260812 08:00:36.147917 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.147917 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.151815 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.160104 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.160233 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.169276 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.183961 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.184039 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.193215 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.193348 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.201762 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.212824 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.216130 27532 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:36.216370 27532 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:36.216390 27532 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:36.216408 27532 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:36.216496 27532 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:36.216634 27532 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:36.216744 27532 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:36.220809 27532 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:36.220914 27532 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:36.220933 27532 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:36.221071 27532 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
W20260812 08:00:36.225265 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.226441 27532 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:33345
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=37235
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.27532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:36.228132 27532 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260812 08:00:36.228343 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.229401 27532 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:36.233047 27532 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:36.234414 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.237851 27541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:36.237845 27542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:36.240482 27544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:36.240957 27532 server_base.cc:1061] running on GCE node
I20260812 08:00:36.241380 27532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:36.242211 27532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:36.243477 27532 hybrid_clock.cc:648] HybridClock initialized: now 1786521636243452 us; error 45 us; skew 500 ppm
I20260812 08:00:36.246414 27532 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
W20260812 08:00:36.246973 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.247541 27532 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:36.247678 27532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:36.251026 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.253307 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.255429 27532 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
W20260812 08:00:36.260421 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.262283 27037 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50112: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:36.263037 27557 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:36.264777 27532 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.003s	sys 0.000s
I20260812 08:00:36.264991 27532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:36.265606 27532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 6
Total live bytes: 6843
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:36.281625 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.284823 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.286902 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.291569 27532 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:36.292368 27532 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:36.292623 27532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:36.293366 27532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:36.295295 27565 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:36.296161 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:36.296178 26982 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:36.296334 26982 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.297128 26982 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.297705 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 12 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.297757 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 12 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:36.298115 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 12 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:36.298556 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 12 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.298755 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 11.
W20260812 08:00:36.299619 24453 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:36.299717 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:36.303176 26982 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:00:36.303612 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 12 candidate_status { last_received { term: 10 index: 78 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.303939 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 10.
I20260812 08:00:36.304410 27532 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:36.304492 27532 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
I20260812 08:00:36.304600 27532 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:36.306870 27532 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:36.306957 27532 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:36.307368 27565 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
W20260812 08:00:36.308645 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.312538 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.318707 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.319386 27157 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:36.319525 27157 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.320273 27157 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.320509 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:36.320657 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:36.320814 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 11.
I20260812 08:00:36.321169 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.321234 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.321301 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.321379 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 11.
I20260812 08:00:36.321430 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 10.
I20260812 08:00:36.321952 26995 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923
I20260812 08:00:36.321982 26999 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:36.322109 26999 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.322677 27157 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260812 08:00:36.322768 27157 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:36.322821 27157 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:00:36.323138 26999 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.323717 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:36.324169 27157 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.324947 27157 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.325249 27088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:36.325443 27088 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 12: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:36.325950 27472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:36.325994 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:36.326125 27472 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 10 FOLLOWER]: Advancing to term 12
W20260812 08:00:36.326435 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.326731 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:36.327147 26995 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:36.327726 27532 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
I20260812 08:00:36.328641 27472 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 12.
I20260812 08:00:36.328989 27532 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:36.329506 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 27532
I20260812 08:00:36.330157 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:36.330188 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.330302 24528 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:00:36.330335 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 11.
I20260812 08:00:36.330497 27688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:36.330570 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:36.331378 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:36.331498 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:00:36.331627 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 12.
W20260812 08:00:36.334656 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.335381 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:36.335536 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:00:36.336525 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 12.
I20260812 08:00:36.337481 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 12.
I20260812 08:00:36.337888 26992 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923
I20260812 08:00:36.338105 27157 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Leader election won for term 12
I20260812 08:00:36.338244 26664 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:36.338344 27157 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 LEADER]: Becoming Leader. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Running, Role: LEADER
W20260812 08:00:36.334656 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.338449 26999 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260812 08:00:36.338580 27565 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:36.338737 27157 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 11.82, Last appended by leader: 82, Current term: 12, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.342518 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b reported cstate change: term changed from 11 to 12, leader changed from 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135) to b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134). New cstate: current_term: 12 leader_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
W20260812 08:00:36.346843 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.353874 27689 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:36.354532 27689 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:36.353437 27637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:36.355274 27689 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:36.354573 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:36.356086 26995 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 12 election: Tablet error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:36.356557 26664 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 12 pre-election: Tablet error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:36.357692 24058 ts_manager.cc:194] Re-registered known tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
W20260812 08:00:36.357870 27347 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 37ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.357903 27157 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:36.357937 27157 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:36.358783 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:36253
I20260812 08:00:36.359418 27472 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 10 index: 78. Preceding OpId from leader: term: 12 index: 84. (index mismatch)
E20260812 08:00:36.360543 27157 fault_injection.cc:59] injecting fault for kudu (pid 26933): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:36.373926 25211 connection.cc:573] client connection to 127.22.208.134:44711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:36.381006 23362 ts_itest-base.cc:575] TS 5 appears to have crashed. Restarting.
I20260812 08:00:36.381497 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:44711
--local_ip_for_outbound_sockets=127.22.208.134
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37749
--webserver_interface=127.22.208.134
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:36.382790 27689 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:36.386119 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.388573 27565 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap replayed 1/1 log segments. Stats: ops{read=82 overwritten=0 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 7 replicates
W20260812 08:00:36.389403 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.389480 27565 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap complete.
I20260812 08:00:36.389964 27565 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.083s	user 0.049s	sys 0.005s
W20260812 08:00:36.392584 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.392715 27565 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Replica starting. Triggering 7 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.395732 27565 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:36.396355 27565 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 11.82, Last appended by leader: 82, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.397533 27565 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260812 08:00:36.401298 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.407442 27690 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:36.408098 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:36.408260 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.409099 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.408814 27563 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20260812 08:00:36.409446 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:36.409655 24528 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 12: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
W20260812 08:00:36.410742 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.410729 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.410919 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 12: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:36.411330 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.411513 27274 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 12: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:36.417786 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:36.418067 27635 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 11.
W20260812 08:00:36.418097 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.419473 24305 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:36.419574 24305 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923; no voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:36.419603 27472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.419796 26940 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 11 FOLLOWER]: Advancing to term 12
I20260812 08:00:36.419816 27472 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 in current term 12: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:36.420969 26940 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260812 08:00:36.424471 27570 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.424804 27586 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.424471 27585 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.424463 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.441655 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.453536 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.462360 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.463169 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.464306 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.465790 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.480623 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.491964 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.499425 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.502508 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.509115 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.509562 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.534190 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.544695 27698 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:36.544992 27698 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:36.545028 27698 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:36.545095 27698 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:36.545192 27698 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:36.545266 27698 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:36.545379 27698 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:36.550077 27698 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:36.550331 27698 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:36.550428 27698 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:36.550611 27698 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
W20260812 08:00:36.551004 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.555843 27698 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:44711
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=37749
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.27698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:36.557102 27698 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:36.558429 27698 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:36.558946 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
E20260812 08:00:36.562063 27698 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:36.566854 27715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:36.566797 27712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:36.567065 27711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:36.567512 27698 server_base.cc:1061] running on GCE node
I20260812 08:00:36.567950 27698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:36.568632 27698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:36.569854 27698 hybrid_clock.cc:648] HybridClock initialized: now 1786521636569829 us; error 48 us; skew 500 ppm
I20260812 08:00:36.573027 27698 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:36.573889 27698 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:36.573995 27698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:36.577329 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.578649 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.579138 27698 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:00:36.587625 27729 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:36.589403 27698 fs_manager.cc:730] Time spent opening block manager: real 0.009s	user 0.004s	sys 0.000s
I20260812 08:00:36.589634 27698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:36.590131 27698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 6
Total live bytes: 6843
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:36.591665 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.593875 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.607582 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.618580 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.623700 27698 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:36.624733 27698 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:36.624972 27698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:36.625870 27698 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:36.627561 27737 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:36.632663 27698 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:36.632766 27698 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20260812 08:00:36.632869 27698 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:36.634781 27698 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:36.634910 27698 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:36.635265 27737 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
W20260812 08:00:36.639042 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.643674 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.649670 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.652922 27698 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:36.652956 27861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
I20260812 08:00:36.654171 27698 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:36.657850 27737 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:36.662235 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 27698
W20260812 08:00:36.664166 24328 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.665006 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.666146 27862 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:36.666455 27862 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:36.667029 27862 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:36.668825 24050 ts_manager.cc:194] Re-registered known tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:36.670064 24050 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:56525
W20260812 08:00:36.674109 24332 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51562: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.683072 27862 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:36.686548 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.687657 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.690922 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.700764 27737 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=6 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20260812 08:00:36.701431 27737 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap complete.
W20260812 08:00:36.701668 27737 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 37ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.709631 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.710801 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.714891 27868 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:36.715057 27868 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.716725 27868 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:36.719684 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.723512 27700 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:36.723658 27700 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.724241 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:36.724354 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.725081 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.725245 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:36.725458 24376 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for term 13 because replica has last-logged OpId of term: 11 index: 82, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 76.
I20260812 08:00:36.725582 27700 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:36.725706 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.725847 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 12.
I20260812 08:00:36.725841 27472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.726013 27472 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for term 13 because replica has last-logged OpId of term: 10 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 76.
I20260812 08:00:36.726725 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.726791 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:36.726866 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 12.
I20260812 08:00:36.726924 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 12.
I20260812 08:00:36.727792 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.727957 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 12.
I20260812 08:00:36.728808 24305 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:36.729115 26940 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260812 08:00:36.729182 26940 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:36.729235 26940 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.729236 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:36.729382 27635 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 11.
I20260812 08:00:36.730701 26940 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.731393 26940 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 13 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:36.733386 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.733701 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.734340 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:36.734504 24528 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.734668 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:36.734972 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.734957 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:36.735093 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.735867 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 13.
I20260812 08:00:36.736034 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 13.
I20260812 08:00:36.736335 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 13.
W20260812 08:00:36.737004 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.737214 27637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:36.737270 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.737418 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 13: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:36.737399 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:36.737457 27637 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for term 13 because replica has last-logged OpId of term: 11 index: 82, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 76.
I20260812 08:00:36.737226 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:36.737835 27473 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.739038 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 13.
I20260812 08:00:36.739153 27737 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.104s	user 0.055s	sys 0.005s
I20260812 08:00:36.740335 24302 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f
I20260812 08:00:36.740624 26940 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 FOLLOWER]: Leader election won for term 13
I20260812 08:00:36.740883 26940 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 LEADER]: Becoming Leader. State: Replica: ada36b19ad0e4386b68527e1ab0ce923, State: Running, Role: LEADER
I20260812 08:00:36.739568 27792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:36.741240 26940 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 11.82, Last appended by leader: 82, Current term: 13, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:36.743100 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 73ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.743279 27737 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.744397 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:36.744582 26747 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
W20260812 08:00:36.745877 27192 leader_election.cc:343] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 13 pre-election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:36.746044 27192 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: f29b76cef775415dbd2c0997fadb3904; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:36.746467 27868 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260812 08:00:36.746668 27562 leader_election.cc:400] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Vote denied by peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.746759 27562 leader_election.cc:403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260812 08:00:36.747251 27737 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
I20260812 08:00:36.747484 27700 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 11 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.747759 24050 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 reported cstate change: term changed from 12 to 13, leader changed from b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134) to ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130). New cstate: current_term: 13 leader_uuid: "ada36b19ad0e4386b68527e1ab0ce923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:36.747853 27737 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 12.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:36.748893 27737 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20260812 08:00:36.750358 27700 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.753818 27792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:36.754105 27792 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 13 because replica has last-logged OpId of term: 12 index: 84, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 82.
I20260812 08:00:36.754462 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 13 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:36.754616 27789 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 12 FOLLOWER]: Advancing to term 13
I20260812 08:00:36.755987 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 13 candidate_status { last_received { term: 9 index: 76 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:36.756166 24528 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 13: Already voted for candidate ada36b19ad0e4386b68527e1ab0ce923 in this term.
I20260812 08:00:36.757050 27789 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 13 because replica has last-logged OpId of term: 12 index: 84, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 82.
I20260812 08:00:36.757843 27864 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:36.758874 27735 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20260812 08:00:36.760062 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:36.760378 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:36.760442 27585 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:36.760560 24528 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.760955 27562 leader_election.cc:400] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.761240 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:36.761428 27274 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.761889 27558 leader_election.cc:400] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Vote denied by peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.761981 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:36.762136 27789 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.762399 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:36.762512 27562 leader_election.cc:400] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Vote denied by peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.762621 27473 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
I20260812 08:00:36.763111 27558 leader_election.cc:400] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f for earlier term 12. Current term is 13.
W20260812 08:00:36.764150 27585 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:36.764956 24376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 12 candidate_status { last_received { term: 11 index: 82 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
W20260812 08:00:36.777820 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.779280 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.783545 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.805580 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.806991 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.807809 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.819640 27892 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:36.819754 27892 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:36.820531 27473 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 10 index: 78. Preceding OpId from leader: term: 13 index: 83. (index mismatch)
I20260812 08:00:36.821200 27892 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:36.825465 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 11 index: 82. Preceding OpId from leader: term: 13 index: 83. (index mismatch)
W20260812 08:00:36.826325 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.826853 27789 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 12 index: 84. Preceding OpId from leader: term: 13 index: 84. (term mismatch)
I20260812 08:00:36.826946 27789 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting all ops after (but not including) 83
I20260812 08:00:36.827035 27789 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted WRITE_OP operation due to leader change: 12.84
I20260812 08:00:36.828346 27274 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 9 index: 76. Preceding OpId from leader: term: 13 index: 84. (index mismatch)
I20260812 08:00:36.829608 26940 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:36.829891 27901 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:36.830071 27901 consensus_queue.cc:1243] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Peer b6d90cd6408d4dc3b47ebf57ca72e91b log is divergent from this leader: its last log entry 12.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 75
I20260812 08:00:36.830377 27902 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:36.831888 27789 raft_consensus.cc:1217] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Deduplicated request from leader. Original: 7.75->[9.76-13.84]   Dedup: 11.82->[13.83-13.84]
I20260812 08:00:36.832089 27789 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting all ops after (but not including) 82
I20260812 08:00:36.832132 27789 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted NO_OP operation due to leader change: 12.83
I20260812 08:00:36.832170 27789 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260812 08:00:36.836544 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 89ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.837276 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 82ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.838624 24528 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 10 index: 78. Preceding OpId from leader: term: 13 index: 84. (index mismatch)
W20260812 08:00:36.838716 26842 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 48ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.839097 27635 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Refusing update from remote peer ada36b19ad0e4386b68527e1ab0ce923: Log matching property violated. Preceding OpId in replica: term: 11 index: 82. Preceding OpId from leader: term: 13 index: 84. (index mismatch)
W20260812 08:00:36.841686 26868 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 58ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.842211 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 4ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.843298 27892 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:36.843698 27892 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 75, Time since last communication: 0.000s
W20260812 08:00:36.852273 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.852317 27699 mvcc.cc:204] Tried to move back new op lower bound from 7317592619393683456 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592619296235520}]
W20260812 08:00:36.857401 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.857998 27893 mvcc.cc:204] Tried to move back new op lower bound from 7317592624428199936 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592619296235520}]
W20260812 08:00:36.861624 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 16ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.861884 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:36.864104 24326 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:51562 (request call id 114) took 1192 ms. Trace:
I20260812 08:00:36.864204 24326 rpcz_store.cc:276] 0812 08:00:35.671785 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:35.671876 (+    91us) service_pool.cc:224] Handling call
0812 08:00:35.672053 (+   177us) tablet_service.cc:2890] Created scanner 82c533ad81024b8382c947d7e95cd6a6 for tablet a12d50650402488683716909cc273d1e, query id is cfb00f1c05764247815c94a767d5c597
0812 08:00:35.672620 (+   567us) tablet_service.cc:3030] Creating iterator
0812 08:00:35.672724 (+   104us) tablet_service.cc:3408] Waiting safe time to advance
0812 08:00:36.007500 (+334776us) tablet_service.cc:3415] Waiting for operations to commit
0812 08:00:36.862172 (+854672us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1189437 microseconds
0812 08:00:36.862260 (+    88us) tablet_service.cc:3055] Iterator created
0812 08:00:36.863528 (+  1268us) tablet_service.cc:3077] Iterator init: OK
0812 08:00:36.863557 (+    29us) tablet_service.cc:3120] has_more: true
0812 08:00:36.863602 (+    45us) tablet_service.cc:3137] Continuing scan request
0812 08:00:36.863663 (+    61us) tablet_service.cc:3201] Found scanner 82c533ad81024b8382c947d7e95cd6a6 for tablet a12d50650402488683716909cc273d1e, query id is cfb00f1c05764247815c94a767d5c597
0812 08:00:36.864089 (+   426us) inbound_call.cc:177] Queueing success response
Metrics: {"cfile_cache_hit":9,"cfile_cache_hit_bytes":1683,"delta_iterators_relevant":1,"rowset_iterators":2,"scanner_bytes_read":439}
W20260812 08:00:36.864959 25323 scanner-internal.cc:458] Time spent opening tablet: real 1.556s	user 0.001s	sys 0.001s
W20260812 08:00:36.866856 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:36.867368 24431 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:36.870772 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.872350 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:36.874300 27585 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.875123 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:36.878244 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.006s	user 0.002s	sys 0.000s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:36.878685 24587 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:36.879475 27864 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:36.880401 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.883579 27909 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 53ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.883930 27690 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
E20260812 08:00:36.885311 27943 fault_injection.cc:59] injecting fault for kudu (pid 24286): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:36.893271 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 39ms of latency in SegmentAllocator::Sync()
I20260812 08:00:36.897671 27563 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.014s	user 0.005s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":224,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:36.898172 27690 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:36.899835 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.900955 27910 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 110ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.906183 26664 connection.cc:573] server connection from 127.22.208.130:38019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:36.906955 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.908561 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.910351 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.910351 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.910462 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.911402 23362 ts_itest-base.cc:575] TS 1 appears to have crashed. Restarting.
I20260812 08:00:36.911878 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:36403
--local_ip_for_outbound_sockets=127.22.208.130
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=33433
--webserver_interface=127.22.208.130
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:36.916723 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.917119 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.918009 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.919287 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.920104 27386 connection.cc:466] server connection from 127.22.208.130:52845 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.130:52845 (request call id 5) could send its response
W20260812 08:00:36.926537 27586 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.926681 27189 connection.cc:466] server connection from 127.22.208.130:44579 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.130:44579 (request call id 5) could send its response
W20260812 08:00:36.926887 27585 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.926558 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.928866 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.934394 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.935442 27735 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.056s	user 0.006s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:36.935766 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.936077 27864 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:36.937636 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 81ms of latency in SegmentAllocator::Sync()
W20260812 08:00:36.938408 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.938853 24454 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.058s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":2,"cfile_cache_miss_bytes":118,"cfile_init":1,"delete_count":35,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":142,"lbm_reads_lt_1ms":6,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":120,"threads_started":1,"update_count":0}
W20260812 08:00:36.943204 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.943207 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.949689 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.949803 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.950659 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.951383 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.952864 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.954198 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.956156 27750 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:36.963408 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:36.969610 25324 meta_cache.cc:1510] marking tablet server ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403) as failed
W20260812 08:00:36.979764 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.980813 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.981916 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.987469 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.989612 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:36.994457 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:36.995538 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.000051 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.004989 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.006093 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.009449 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.012436 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.014361 27563 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.116s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":326,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:37.015702 27690 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
W20260812 08:00:37.016587 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.017338 27752 tablet_service.cc:3080] error setting up scanner 56c657c048b24defb86253bb7467b155 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.183s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
I20260812 08:00:37.020303 27735 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.084s	user 0.001s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":118,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:37.020730 27864 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
W20260812 08:00:37.020776 27910 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 48ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.025206 27752 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.025230 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.026397 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 38ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.031509 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.032553 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.035481 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.040086 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.043584 27751 tablet_service.cc:3080] error setting up scanner 96d9a5cf01dd42e4af2ad42f083b2ca1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.209s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
W20260812 08:00:37.044493 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.048506 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.052386 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.054514 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.055820 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.057482 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.063265 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.065742 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.067073 27735 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.046s	user 0.006s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":4,"cfile_cache_miss_bytes":171,"cfile_init":2,"delete_count":35,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":131,"lbm_read_time_us":193,"lbm_reads_lt_1ms":12,"lbm_write_time_us":1021,"lbm_writes_lt_1ms":4,"mutex_wait_us":2,"reinsert_count":0,"spinlock_wait_cycles":3328,"update_count":0}
W20260812 08:00:37.071763 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.073475 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.074709 27563 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.057s	user 0.006s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":4,"cfile_cache_miss_bytes":171,"cfile_init":2,"delete_count":35,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":1354,"lbm_read_time_us":180,"lbm_reads_lt_1ms":12,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":237,"threads_started":2,"update_count":0}
W20260812 08:00:37.078926 27946 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:37.079196 27946 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:37.079241 27946 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:37.079275 27946 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:37.079351 27946 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:37.079416 27946 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:37.079515 27946 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:37.080981 27751 tablet_service.cc:3080] error setting up scanner 7ed74fe5744b4a2db821d02aea692a27 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.247s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
W20260812 08:00:37.084462 27946 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:37.084558 27946 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:37.084576 27946 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:37.084630 27946 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
W20260812 08:00:37.087579 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.090423 27946 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:36403
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=33433
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.27946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:37.092082 27946 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:37.093557 27946 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:37.095312 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
E20260812 08:00:37.097795 27946 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:37.101876 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.103804 27954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:37.103842 27957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:37.103814 27955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:37.104724 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:37.105296 27946 server_base.cc:1061] running on GCE node
I20260812 08:00:37.105855 27946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:37.106675 27946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:37.107704 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.108614 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.108281 27946 hybrid_clock.cc:648] HybridClock initialized: now 1786521637108226 us; error 72 us; skew 500 ppm
W20260812 08:00:37.112082 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.112599 27946 webserver.cc:533] Webserver started at http://127.22.208.130:33433/ using document root <none> and password file <none>
I20260812 08:00:37.113287 27946 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:37.113370 27946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:37.114720 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.115382 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.121888 27946 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.003s	sys 0.000s
W20260812 08:00:37.127058 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.128087 27751 tablet_service.cc:3080] error setting up scanner 50cfe29102f84b31bb041f1c7aa92175 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.294s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
I20260812 08:00:37.131810 27971 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:37.132927 27896 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.133086 27896 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.133303 27946 fs_manager.cc:730] Time spent opening block manager: real 0.009s	user 0.001s	sys 0.002s
I20260812 08:00:37.133442 27946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "ada36b19ad0e4386b68527e1ab0ce923"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:37.134006 27946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 7
Total live bytes: 7055
Total live bytes (after alignment): 32768
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:37.134918 27896 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:37.136349 27868 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.136514 27868 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.137459 27868 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:37.137601 24474 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.138320 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.140142 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.140316 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:37.140321 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.140494 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 13.
W20260812 08:00:37.140565 27191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.141207 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.141309 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:37.141415 26747 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 for term 14 because replica has last-logged OpId of term: 13 index: 85, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 84.
I20260812 08:00:37.142225 26999 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.142343 26999 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:37.142860 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:37.143213 26999 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.143724 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.143877 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 13.
I20260812 08:00:37.144027 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:37.145052 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.145226 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 13.
W20260812 08:00:37.148099 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:37.148666 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:37.149330 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:37.149688 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:37.150447 27387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.152662 27915 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.152967 27915 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.153033 27192 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:37.154223 27868 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260812 08:00:37.154279 27915 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.155082 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.155253 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 13.
I20260812 08:00:37.155583 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.155738 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 13.
W20260812 08:00:37.156217 27387 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.157012 27933 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.157159 27933 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.157971 27933 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.157927 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.158135 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 13.
I20260812 08:00:37.158494 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:37.158670 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 13.
I20260812 08:00:37.159373 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:37.159447 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.159584 27789 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 13.
W20260812 08:00:37.159955 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.161154 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.161302 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 
W20260812 08:00:37.161737 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.162103 27915 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260812 08:00:37.162091 27275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.162164 27915 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
W20260812 08:00:37.162160 27191 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.162210 27915 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:00:37.162256 27275 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 13.
I20260812 08:00:37.162781 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.162915 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 13.
I20260812 08:00:37.163146 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.163290 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 13.
I20260812 08:00:37.163587 27915 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.163678 27558 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:37.164517 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.164688 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 13.
W20260812 08:00:37.165508 27561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
W20260812 08:00:37.165702 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.166061 27933 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260812 08:00:37.166126 27933 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Starting leader election (detected failure of leader ada36b19ad0e4386b68527e1ab0ce923)
I20260812 08:00:37.166172 27933 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:00:37.167116 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:37.167261 27473 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:00:37.167452 27933 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.168301 27933 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.168466 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 14.
I20260812 08:00:37.169492 27275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:37.169626 27275 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 13 FOLLOWER]: Advancing to term 14
W20260812 08:00:37.169835 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.170156 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:37.170287 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:00:37.170315 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:37.170432 27789 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 13 FOLLOWER]: Advancing to term 14
I20260812 08:00:37.170857 27275 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 14.
I20260812 08:00:37.171396 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.171674 27789 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.171667 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:37.171818 27274 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 14: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:37.173000 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:37.173188 24529 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 14: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
W20260812 08:00:37.174500 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.174482 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:37.174671 27473 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 14: Already voted for candidate 9dede2273d3a4266a1bf7a5b368080f8 in this term.
I20260812 08:00:37.176069 27915 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 election: Requested vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.176097 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:37.176280 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:37.177112 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:37.177213 27561 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.177335 27789 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
W20260812 08:00:37.178088 27561 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.178170 27561 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:37.178395 27933 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260812 08:00:37.179579 27792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:37.179778 27792 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:37.179793 27946 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:37.180657 27946 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:37.180894 27946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:37.181224 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:37.181401 27789 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:37.181713 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.181928 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3a0fdc906c74926946b47c61bd5cea5 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:37.182125 27946 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:37.182439 27388 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:37.182806 27896 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260812 08:00:37.184630 27997 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:37.185084 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:37.185282 27789 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
W20260812 08:00:37.185910 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.190022 27637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:37.190150 27633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 14 candidate_status { last_received { term: 13 index: 84 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:37.189966 27635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 14 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:37.190231 27637 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 14: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
W20260812 08:00:37.190529 26663 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.190779 26663 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
I20260812 08:00:37.190996 26999 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260812 08:00:37.191450 24452 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
W20260812 08:00:37.191555 24452 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:37.191984 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:37.192135 27946 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:37.192203 27946 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.000s	sys 0.000s
I20260812 08:00:37.192190 27915 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260812 08:00:37.192279 27946 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
W20260812 08:00:37.192368 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.194639 27946 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:37.194726 27946 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.002s
I20260812 08:00:37.195258 27997 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap starting.
W20260812 08:00:37.196137 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.197806 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.197844 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.211253 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:37.212196 27946 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:36403
I20260812 08:00:37.213256 27946 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
W20260812 08:00:37.215523 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.216499 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.217100 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:37.217612 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 27946
W20260812 08:00:37.218438 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.219991 27997 log.cc:826] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:37.221035 28123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:36403 every 8 connection(s)
W20260812 08:00:37.224274 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.229079 28124 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:37.229476 28124 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:37.230238 28124 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:37.232777 24058 ts_manager.cc:194] Re-registered known tserver with Master: ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403)
I20260812 08:00:37.233799 24058 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:36405
W20260812 08:00:37.244503 27751 tablet_service.cc:3080] error setting up scanner 86f6891ecdba4c3d92244f3dded55527 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.410s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
I20260812 08:00:37.250519 28124 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:37.263607 27997 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=75 ignored=70} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 10 replicates
W20260812 08:00:37.263638 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.263933 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.264454 27997 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap complete.
I20260812 08:00:37.264935 27997 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent bootstrapping tablet: real 0.070s	user 0.054s	sys 0.005s
W20260812 08:00:37.267483 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.268795 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.268924 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.268635 27997 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 FOLLOWER]: Replica starting. Triggering 10 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.271538 27997 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada36b19ad0e4386b68527e1ab0ce923, State: Initialized, Role: FOLLOWER
I20260812 08:00:37.272311 27997 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 13.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.273654 27997 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
W20260812 08:00:37.273722 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.276412 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.283587 28125 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:37.284809 27995 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20260812 08:00:37.291620 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.294394 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.296406 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.299027 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.312745 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.320622 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.323454 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.325349 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.325433 27752 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.325560 27759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.328953 27759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.331658 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.350538 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.352167 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.353641 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.356961 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.375733 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.378561 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.379943 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.381819 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.381851 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.383881 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.388319 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.389249 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.411276 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.411429 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.414155 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.421700 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.435099 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.438921 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.438999 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.443804 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.443895 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.446292 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.447042 27587 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58480: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:37.453722 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.479389 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.479477 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.479477 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.486953 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.494969 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.495370 27933 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:37.495513 27933 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.496419 27933 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.496465 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:37.496627 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.496816 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.497004 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
W20260812 08:00:37.497156 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.498189 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.498283 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.498327 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.498440 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.498474 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.498842 27558 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:37.499012 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:37.499173 27789 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 14.
I20260812 08:00:37.499372 27933 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260812 08:00:37.499456 27933 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:37.499557 27933 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.500751 27933 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.501801 27933 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 15 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.502065 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:37.502210 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.502974 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:37.503090 24529 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.503545 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.504076 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.504329 27789 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
I20260812 08:00:37.504448 27789 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.504571 28054 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:37.504877 28054 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 13 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.505352 27789 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.505756 27562 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:37.505977 27933 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 FOLLOWER]: Leader election won for term 15
I20260812 08:00:37.506186 27933 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 LEADER]: Becoming Leader. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Running, Role: LEADER
I20260812 08:00:37.506884 27933 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 13.85, Last appended by leader: 85, Current term: 15, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.507045 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:37.507170 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.507421 28054 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.508071 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:37.508131 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.508222 27473 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 14 FOLLOWER]: Advancing to term 15
I20260812 08:00:37.509390 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15.
I20260812 08:00:37.504571 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 15 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:37.509776 28071 raft_consensus.cc:2376] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 15. Re-sending same reply.
W20260812 08:00:37.511524 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.514714 24056 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f reported cstate change: term changed from 13 to 15, leader changed from ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130) to 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135). New cstate: current_term: 15 leader_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
W20260812 08:00:37.515241 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.516644 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.518523 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.519945 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.551630 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.553879 27910 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 77ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.553960 27933 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:37.554028 27933 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:37.554760 27473 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 15 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 13 index: 84. Preceding OpId from leader: term: 15 index: 87. (index mismatch)
E20260812 08:00:37.554817 28134 fault_injection.cc:59] injecting fault for kudu (pid 27532): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:37.554867 27789 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 13 index: 85. Preceding OpId from leader: term: 15 index: 87. (index mismatch)
W20260812 08:00:37.567471 27387 connection.cc:573] server connection from 127.22.208.135:52825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:37.567487 27730 connection.cc:573] server connection from 127.22.208.135:42547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:37.568233 25211 connection.cc:573] client connection to 127.22.208.135:33345 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:37.568387 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:37.568782 23362 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 08:00:37.569202 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:33345
--local_ip_for_outbound_sockets=127.22.208.135
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37235
--webserver_interface=127.22.208.135
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:37.569331 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.590113 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.601886 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.604748 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.604779 27223 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.626384 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.628444 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.629668 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.629925 27759 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.632380 27759 tablet_service.cc:3080] error setting up scanner a1b1c4fc381748289f72a8ffd5f600c9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.798s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
W20260812 08:00:37.632740 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.637157 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.640324 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.644843 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.668053 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.673986 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.679499 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.683030 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.685853 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.690982 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.693511 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.694878 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.711472 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.711541 27422 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.717692 28144 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:37.717934 28144 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:37.717953 28144 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:37.717968 28144 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:37.718051 28144 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:37.718103 28144 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:37.718214 28144 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:37.723062 28144 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:37.723222 28144 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:37.723258 28144 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:37.723346 28144 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
W20260812 08:00:37.724718 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.728237 28144 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:33345
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=37235
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.28144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:37.729413 28144 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:37.730705 28144 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:37.734200 28144 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:37.738634 28152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:37.738634 28149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:37.738865 28150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:37.739750 28144 server_base.cc:1061] running on GCE node
I20260812 08:00:37.740234 28144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:37.740876 28144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:37.742110 28144 hybrid_clock.cc:648] HybridClock initialized: now 1786521637742053 us; error 70 us; skew 500 ppm
I20260812 08:00:37.744968 28144 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
I20260812 08:00:37.745805 28144 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:37.745957 28144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:37.749078 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.751447 28144 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
W20260812 08:00:37.754720 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
W20260812 08:00:37.755523 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.761636 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.763305 28165 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:37.764519 28144 fs_manager.cc:730] Time spent opening block manager: real 0.012s	user 0.004s	sys 0.000s
I20260812 08:00:37.764640 28144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:37.765084 28144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:37.779126 28144 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:37.779911 28144 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:37.780141 28144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:37.780833 28144 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:37.783715 28173 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:37.788259 28144 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:37.788338 28144 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20260812 08:00:37.788391 28144 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:37.790042 28144 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:37.790118 28144 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 08:00:37.790422 28173 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
W20260812 08:00:37.795253 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.803443 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.803565 28144 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
W20260812 08:00:37.804754 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.804886 28144 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:37.807556 28304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:37.809795 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 28144
W20260812 08:00:37.813529 27751 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50218: Illegal state: replica b6d90cd6408d4dc3b47ebf57ca72e91b is not leader of this config: current role FOLLOWER
I20260812 08:00:37.814404 28173 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:37.818318 28312 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:37.818486 28312 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.823280 28312 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:37.824096 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.825511 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:37.832640 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:37.832844 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 15.
I20260812 08:00:37.833387 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:37.833417 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:37.833546 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 15.
I20260812 08:00:37.833554 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 15.
I20260812 08:00:37.834039 27730 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:37.834355 28312 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260812 08:00:37.834424 28312 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:37.834481 28312 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.835691 28312 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.836526 28312 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:37.836766 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:37.836862 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:37.836917 28071 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.836992 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.838071 27274 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16.
I20260812 08:00:37.838122 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16.
I20260812 08:00:37.839072 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:37.839203 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.840034 24529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:37.840179 24529 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 15.
I20260812 08:00:37.840318 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16.
I20260812 08:00:37.840564 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:37.840682 24528 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.841835 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16.
I20260812 08:00:37.842715 27472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:37.842856 27472 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 15 FOLLOWER]: Advancing to term 16
I20260812 08:00:37.843915 27472 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16.
I20260812 08:00:37.844137 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:37.844247 27473 raft_consensus.cc:2376] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 16. Re-sending same reply.
I20260812 08:00:37.846925 27732 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:37.847225 28312 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 FOLLOWER]: Leader election won for term 16
I20260812 08:00:37.847508 28312 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 LEADER]: Becoming Leader. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Running, Role: LEADER
I20260812 08:00:37.847877 28312 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 13.85, Last appended by leader: 85, Current term: 16, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:37.849670 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 29ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.851732 28248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:37.851732 28250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 16 candidate_status { last_received { term: 13 index: 85 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
W20260812 08:00:37.855079 27734 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 election: Tablet error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:37.855530 27734 leader_election.cc:343] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 16 pre-election: Tablet error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:37.855479 24055 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b reported cstate change: term changed from 15 to 16, leader changed from 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135) to b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134). New cstate: current_term: 16 leader_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:37.865619 28307 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:37.865917 28307 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:37.866729 28307 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:37.868871 24054 ts_manager.cc:194] Re-registered known tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
I20260812 08:00:37.870123 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:59235
I20260812 08:00:37.873900 28173 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260812 08:00:37.874471 28173 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap complete.
W20260812 08:00:37.874694 28173 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.884183 28307 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:37.898334 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:37.899788 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 2ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.900569 28312 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:37.900640 28312 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:37.901379 28173 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.111s	user 0.065s	sys 0.004s
I20260812 08:00:37.901415 27473 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 13 index: 84. Preceding OpId from leader: term: 16 index: 87. (index mismatch)
I20260812 08:00:37.902055 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 13 index: 85. Preceding OpId from leader: term: 16 index: 87. (index mismatch)
I20260812 08:00:37.902760 27274 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 13 index: 84. Preceding OpId from leader: term: 16 index: 87. (index mismatch)
I20260812 08:00:37.903439 28312 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:37.903769 28326 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:37.903715 28173 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:37.904220 28312 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:37.905994 28173 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:37.906646 28173 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 15.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:37.907037 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 99ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.907845 28173 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:00:37.911676 28071 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 13 index: 85. Preceding OpId from leader: term: 16 index: 87. (index mismatch)
I20260812 08:00:37.911778 28308 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:37.912341 24528 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 13 index: 85. Preceding OpId from leader: term: 16 index: 87. (index mismatch)
I20260812 08:00:37.913264 28170 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20260812 08:00:37.913836 28350 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56354 (ReqId={client: d4f5c21089f84a189aa50e9d87bda159, seq_no=75, attempt_no=17}) took 2217 ms. Trace:
I20260812 08:00:37.913928 28350 rpcz_store.cc:276] 0812 08:00:35.696323 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:35.696394 (+    71us) service_pool.cc:224] Handling call
0812 08:00:37.913811 (+2217417us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260812 08:00:37.914605 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 73ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.914894 28248 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 15 FOLLOWER]: Advancing to term 16
W20260812 08:00:37.917009 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 26ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.917445 28248 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Refusing update from remote peer b6d90cd6408d4dc3b47ebf57ca72e91b: Log matching property violated. Preceding OpId in replica: term: 15 index: 87. Preceding OpId from leader: term: 16 index: 87. (term mismatch)
I20260812 08:00:37.917521 28248 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting all ops after (but not including) 86
I20260812 08:00:37.917585 28248 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting uncommitted WRITE_OP operation due to leader change: 15.87
I20260812 08:00:37.918727 28329 mvcc.cc:204] Tried to move back new op lower bound from 7317592624428199936 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592619296235520}]
I20260812 08:00:37.922850 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:37.923785 27334 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:37.924029 28312 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20260812 08:00:37.924180 28312 consensus_queue.cc:1243] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Peer 3e1efe84be134f1bb72a09c4c17aa52f log is divergent from this leader: its last log entry 15.86 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 84
I20260812 08:00:37.924264 28349 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 75, Time since last communication: 0.000s
I20260812 08:00:37.924691 28349 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20260812 08:00:37.928639 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:37.929214 27909 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 91ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.933216 28354 mvcc.cc:204] Tried to move back new op lower bound from 7317592624428199936 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592619296235520 or (T in {7317592619390918656})}]
I20260812 08:00:37.934383 28248 raft_consensus.cc:1217] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Deduplicated request from leader. Original: 13.84->[13.85-16.89]   Dedup: 13.85->[16.86-16.89]
I20260812 08:00:37.934729 28248 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting all ops after (but not including) 85
W20260812 08:00:37.935009 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 97ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.935212 28342 mvcc.cc:204] Tried to move back new op lower bound from 7317592624428199936 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592624428199936 or (T in {7317592624428199936})}]
I20260812 08:00:37.934792 28248 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting uncommitted NO_OP operation due to leader change: 15.86
I20260812 08:00:37.936995 28248 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:37.938881 28365 mvcc.cc:204] Tried to move back new op lower bound from 7317592624428199936 to 7317592617154895872. Current Snapshot: MvccSnapshot[applied={T|T < 7317592619296235520}]
W20260812 08:00:37.941753 28368 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 70ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.943704 27905 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 133ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.952219 28378 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 6ms of latency in SegmentAllocator::Sync()
I20260812 08:00:37.957957 28125 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:37.964896 28378 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 55ms of latency in SegmentAllocator::Sync()
W20260812 08:00:37.988281 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 27ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.006927 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 183ms of latency in SegmentAllocator::Sync()
E20260812 08:00:38.007925 28326 fault_injection.cc:59] injecting fault for kudu (pid 27698): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:38.013062 28368 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.016081 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 92ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.018409 27195 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.094s	user 0.004s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":199,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.019052 27334 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:38.026814 27989 connection.cc:573] server connection from 127.22.208.134:43915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:38.026885 28169 connection.cc:573] server connection from 127.22.208.134:56603 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:38.027268 24453 connection.cc:573] server connection from 127.22.208.134:57521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:38.027369 25324 meta_cache.cc:1510] marking tablet server b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) as failed
W20260812 08:00:38.032742 26663 connection.cc:466] server connection from 127.22.208.134:44757 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.134:44757 (request call id 3) could send its response
I20260812 08:00:38.033931 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.111s	user 0.001s	sys 0.000s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":157,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.034332 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:38.036554 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 111ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.036926 23362 ts_itest-base.cc:575] TS 5 appears to have crashed. Restarting.
I20260812 08:00:38.037395 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:44711
--local_ip_for_outbound_sockets=127.22.208.134
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37749
--webserver_interface=127.22.208.134
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:38.037900 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:38.090081 27995 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.132s	user 0.004s	sys 0.000s Metrics: {"bytes_written":212,"delete_count":5,"lbm_write_time_us":238,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.090547 28125 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:38.093302 28368 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 115ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.095721 28012 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.095794 28011 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.096132 28007 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.096364 28009 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.108560 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 20ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.108770 27190 connection.cc:466] server connection from 127.22.208.134:60113 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.134:60113 (request call id 4) could send its response
I20260812 08:00:38.130487 27195 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.111s	user 0.001s	sys 0.001s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":334,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.131135 27334 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
I20260812 08:00:38.136739 27195 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.005s	user 0.005s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":4,"cfile_cache_miss_bytes":171,"cfile_init":2,"delete_count":35,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":247,"lbm_read_time_us":175,"lbm_reads_lt_1ms":12,"lbm_write_time_us":128,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":212,"threads_started":2,"update_count":0}
I20260812 08:00:38.149796 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.115s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":2,"cfile_cache_miss_bytes":118,"cfile_init":1,"delete_count":35,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":51,"lbm_reads_lt_1ms":6,"lbm_write_time_us":97,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":123,"threads_started":1,"update_count":0}
W20260812 08:00:38.162377 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.166262 28379 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:38.166654 28379 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:38.166831 28379 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:38.166936 28379 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:38.167013 28379 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:38.167057 28379 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:38.167143 28379 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:38.171794 28379 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:38.171895 28379 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:38.171912 28379 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:38.172032 28379 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
I20260812 08:00:38.176834 28379 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:44711
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=37749
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.28379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:38.178099 28379 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:38.179574 28379 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:38.183236 28379 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:38.188030 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.188040 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.188125 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:38.188501 28379 server_base.cc:1061] running on GCE node
I20260812 08:00:38.188874 28379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:38.189590 28379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:38.190263 27391 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.256s	user 0.000s	sys 0.000s
I20260812 08:00:38.190819 28379 hybrid_clock.cc:648] HybridClock initialized: now 1786521638190804 us; error 39 us; skew 500 ppm
I20260812 08:00:38.192351 27391 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.264s	user 0.004s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":182,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.192914 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:38.194401 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 13ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.194762 28379 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:38.195411 28379 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:38.195473 28379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:38.199640 28379 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
I20260812 08:00:38.205926 28328 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.206101 28328 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.207010 28328 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:38.207692 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.208668 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.208814 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.208923 28409 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:38.209062 27391 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.016s	user 0.002s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":134,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:38.210167 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
I20260812 08:00:38.211131 28379 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.003s	sys 0.000s
I20260812 08:00:38.211284 28379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:38.211982 28379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:38.212420 27995 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.122s	user 0.005s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":2,"cfile_cache_miss_bytes":118,"cfile_init":1,"delete_count":35,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":109,"lbm_reads_lt_1ms":6,"lbm_write_time_us":153,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":236,"threads_started":2,"update_count":0}
W20260812 08:00:38.215833 27388 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.215960 27388 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: e3a0fdc906c74926946b47c61bd5cea5; no voters: 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.216156 28328 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 08:00:38.218255 28248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:38.222349 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.226401 27391 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.016s	user 0.007s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":431,"cfile_cache_miss":4,"cfile_cache_miss_bytes":171,"cfile_init":2,"delete_count":35,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":290,"lbm_read_time_us":192,"lbm_reads_lt_1ms":12,"lbm_write_time_us":3890,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":3,"reinsert_count":0,"thread_start_us":246,"threads_started":2,"update_count":0}
I20260812 08:00:38.228447 28363 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.228663 28363 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.229735 28363 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.230670 28370 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.230827 28370 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.231369 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.231748 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.232889 28370 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.233129 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.233284 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 16.
W20260812 08:00:38.240864 27991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.242791 28379 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:38.243805 28379 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:38.244091 28379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:38.244920 28379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:38.248296 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.251618 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.251871 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 16.
I20260812 08:00:38.253522 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.254026 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 16.
I20260812 08:00:38.255002 28248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:38.256496 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:38.256557 27991 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.256850 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.257021 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 16.
I20260812 08:00:38.257073 27988 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.257534 28370 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 08:00:38.258548 28248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 17 candidate_status { last_received { term: 16 index: 89 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:38.258659 28438 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:38.259131 24453 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.259353 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.259553 28363 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 08:00:38.265012 28379 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:38.265210 28379 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.017s	user 0.001s	sys 0.000s
I20260812 08:00:38.265304 28379 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:38.265360 28340 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.265564 28340 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.267532 28340 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.267716 28379 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:38.267812 28379 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:38.274814 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.275010 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 16.
W20260812 08:00:38.275899 26679 tablet_service.cc:3080] error setting up scanner f1b26257e3ab4028a3da3586d83cb494 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.366s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
I20260812 08:00:38.277978 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:38.279394 28169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.279860 28438 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
I20260812 08:00:38.280226 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.280392 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 16.
I20260812 08:00:38.282807 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.282999 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.283152 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 16.
I20260812 08:00:38.283524 28166 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.283823 28340 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260812 08:00:38.283893 28340 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Starting leader election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.283937 28340 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.285091 28340 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.285943 28340 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.286029 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:38.286185 24528 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.286496 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:38.286665 28071 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.286664 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:38.287257 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:38.287371 27473 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.287454 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 17.
I20260812 08:00:38.287763 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 17.
I20260812 08:00:38.288456 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 17.
I20260812 08:00:38.289302 28166 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 
I20260812 08:00:38.289440 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 17 candidate_status { last_received { term: 16 index: 94 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:38.289575 28340 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Leader election won for term 17
I20260812 08:00:38.289839 28340 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 LEADER]: Becoming Leader. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Running, Role: LEADER
I20260812 08:00:38.290190 28340 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 16.94, Last appended by leader: 94, Current term: 17, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:38.292356 28378 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 98ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.294129 24056 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f reported cstate change: term changed from 16 to 17, leader changed from b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134) to 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135). New cstate: current_term: 17 leader_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
W20260812 08:00:38.295814 28169 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:38.296217 28169 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:38.296330 28379 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:38.297228 28574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
I20260812 08:00:38.297799 28379 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:38.299850 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 28379
I20260812 08:00:38.312290 28438 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:38.312366 28341 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b6d90cd6408d4dc3b47ebf57ca72e91b)
I20260812 08:00:38.312489 28341 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.313553 28341 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.314783 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.315773 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.316161 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.316329 27473 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 17: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:38.318409 24528 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 17: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:38.322505 28575 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:38.322933 28575 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:38.324000 28575 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:38.327090 24056 ts_manager.cc:194] Re-registered known tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:38.328524 24056 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:52631
I20260812 08:00:38.329507 28248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:38.330166 26664 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.330390 28341 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.331787 28341 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260812 08:00:38.335249 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.335489 28071 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 17: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:38.335716 28514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 17 candidate_status { last_received { term: 16 index: 87 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:38.337765 26664 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 17 pre-election: Tablet error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:38.343137 28575 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:38.369798 28438 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=8 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 8 replicates
I20260812 08:00:38.370512 28438 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap complete.
W20260812 08:00:38.370793 28438 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 12ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.381245 28340 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:38.381343 28340 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:38.382061 27274 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.383304 27274 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 95. Preceding OpId from leader: term: 17 index: 95. (term mismatch)
I20260812 08:00:38.383358 28438 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.104s	user 0.065s	sys 0.004s
I20260812 08:00:38.383399 27274 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Aborting all ops after (but not including) 94
I20260812 08:00:38.383471 27274 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Aborting uncommitted WRITE_OP operation due to leader change: 16.95
I20260812 08:00:38.384588 28340 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:38.386006 27906 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 102ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.386044 28438 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 FOLLOWER]: Replica starting. Triggering 8 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.389146 28438 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
I20260812 08:00:38.389678 28438 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 16.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.390637 28438 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260812 08:00:38.401360 28576 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:38.402952 28435 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20260812 08:00:38.407821 27473 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 87. Preceding OpId from leader: term: 17 index: 95. (index mismatch)
I20260812 08:00:38.408416 28340 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:38.411846 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 31ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.412302 28071 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 89. Preceding OpId from leader: term: 17 index: 95. (index mismatch)
I20260812 08:00:38.413540 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 87. Preceding OpId from leader: term: 17 index: 95. (index mismatch)
I20260812 08:00:38.413501 28588 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:38.416564 28368 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 84ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.418275 28340 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
I20260812 08:00:38.419831 24528 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 89. Preceding OpId from leader: term: 17 index: 95. (index mismatch)
I20260812 08:00:38.420282 28514 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 16 FOLLOWER]: Advancing to term 17
I20260812 08:00:38.421030 28340 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:38.422436 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 32ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.422837 28514 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 16 index: 92. Preceding OpId from leader: term: 17 index: 95. (index mismatch)
W20260812 08:00:38.423784 27909 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 115ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.424469 28340 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:38.427716 28589 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 91ms of latency in SegmentAllocator::Sync()
W20260812 08:00:38.452841 27224 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52666: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
E20260812 08:00:38.489161 28340 fault_injection.cc:59] injecting fault for kudu (pid 28144): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:38.501138 27991 connection.cc:466] server connection from 127.22.208.135:59637 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:59637 (request call id 3) could send its response
I20260812 08:00:38.504383 23362 ts_itest-base.cc:575] TS 6 appears to have crashed. Restarting.
I20260812 08:00:38.504909 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:33345
--local_ip_for_outbound_sockets=127.22.208.135
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37235
--webserver_interface=127.22.208.135
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.05
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:38.519604 28419 connection.cc:466] server connection from 127.22.208.135:36691 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:36691 (request call id 1) could send its response
W20260812 08:00:38.524322 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.539558 24451 connection.cc:466] server connection from 127.22.208.135:36607 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.135:36607 (request call id 3) could send its response
W20260812 08:00:38.596097 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.654791 28590 flags.cc:432] Enabled unsafe flag: --fault_crash_on_leader_request_fraction=0.050000000000000003
W20260812 08:00:38.655196 28590 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:38.655259 28590 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:38.655297 28590 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:38.655365 28590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:38.655412 28590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:38.655499 28590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:38.659612 28590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:38.659706 28590 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:38.659724 28590 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:38.659870 28590 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
I20260812 08:00:38.665055 28590 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_crash_on_leader_request_fraction=0.050000000000000003
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:33345
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=37235
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.28590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:38.666499 28590 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:38.668570 28590 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:38.670295 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
E20260812 08:00:38.673004 28590 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:38.678107 28604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.678328 28598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.678107 28599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:38.678658 28590 server_base.cc:1061] running on GCE node
I20260812 08:00:38.679436 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:38.680202 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:38.683823 28590 hybrid_clock.cc:648] HybridClock initialized: now 1786521638683794 us; error 53 us; skew 500 ppm
I20260812 08:00:38.686188 28590 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
I20260812 08:00:38.686971 28590 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:38.687047 28590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:38.691303 28590 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:38.698855 28617 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:38.700074 28590 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.002s	sys 0.001s
I20260812 08:00:38.700197 28590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:38.700644 28590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:38.710058 28328 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.710214 28328 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.710932 28328 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:38.711320 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.711287 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.712646 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
W20260812 08:00:38.713205 27388 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:38.713341 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.713389 27388 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1
I20260812 08:00:38.713553 28328 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260812 08:00:38.714993 28370 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.715173 28370 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.716013 28370 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.716678 28514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:38.717108 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.717499 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.717923 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.717950 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.718071 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 17.
W20260812 08:00:38.719502 27991 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:38.719750 27991 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.720729 28370 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260812 08:00:38.721206 28341 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.721323 28341 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.721884 28341 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.722702 28514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:38.722761 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.722885 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 17.
I20260812 08:00:38.723239 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.723390 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc6ed4943d5493caaf8b5feba124db1 in term 17.
I20260812 08:00:38.723667 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.723798 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.724583 28514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
W20260812 08:00:38.725034 26664 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:38.725119 26664 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, b6d90cd6408d4dc3b47ebf57ca72e91b, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.725277 28341 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260812 08:00:38.726258 28587 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.726524 28587 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.728400 28587 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:38.733109 28421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:38.734138 28590 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:38.735117 28590 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:38.735426 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:38.735843 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.736013 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 17.
I20260812 08:00:38.736295 28590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:38.737538 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.738224 28635 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:38.741714 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.741914 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 17.
W20260812 08:00:38.742313 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:38.743835 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.743866 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:38.744040 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 17.
I20260812 08:00:38.744453 28416 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 9dede2273d3a4266a1bf7a5b368080f8
W20260812 08:00:38.744624 28421 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:38.745009 28587 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260812 08:00:38.745086 28587 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.745143 28587 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:00:38.745623 28590 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:38.745695 28590 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.000s	sys 0.000s
I20260812 08:00:38.745774 28590 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:38.746237 28587 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.747085 28587 leader_election.cc:290] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.747833 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:38.747979 27473 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:00:38.748248 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:38.748380 28071 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:00:38.748804 28590 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:38.748864 28590 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:00:38.749215 27473 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 18.
I20260812 08:00:38.749225 28071 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 18.
I20260812 08:00:38.750144 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:38.750954 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:38.751129 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:00:38.751358 27274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:38.751444 28635 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
I20260812 08:00:38.752346 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in term 18.
I20260812 08:00:38.752754 28421 leader_election.cc:304] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: 9dede2273d3a4266a1bf7a5b368080f8, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:38.752974 28587 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 FOLLOWER]: Leader election won for term 18
I20260812 08:00:38.753268 28587 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 LEADER]: Becoming Leader. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Running, Role: LEADER
I20260812 08:00:38.753638 28587 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 18, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:38.755033 28421 leader_election.cc:336] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:38.756872 28589 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.759235 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b reported cstate change: term changed from 17 to 18, leader changed from 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135) to b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134). New cstate: current_term: 18 leader_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:38.773023 28635 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:38.774425 28590 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
I20260812 08:00:38.775020 28767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:38.775611 28590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:38.781265 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 28590
I20260812 08:00:38.786530 28356 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.786743 28356 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.787679 28356 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.789654 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.789947 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:38.791651 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 27160
I20260812 08:00:38.791996 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.792199 27473 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f29b76cef775415dbd2c0997fadb3904 in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:38.795141 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:38.797084 28768 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:38.797648 28768 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:38.798714 28768 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:38.801245 24055 ts_manager.cc:194] Re-registered known tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
I20260812 08:00:38.802201 24055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:52363
W20260812 08:00:38.807610 26663 connection.cc:573] server connection from 127.22.208.129:39609 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:38.807996 27387 connection.cc:573] server connection from 127.22.208.129:44115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:38.808187 28584 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:00:38.788952 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.22.208.129:38189 (local address 127.22.208.134:44711)
0812 08:00:38.789095 (+   143us) server_negotiation.cc:207] Beginning negotiation
0812 08:00:38.789100 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:00:38.807856 (+ 18756us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.208.129:38189: BlockingRecv error: recv got EOF from 127.22.208.129:38189 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 08:00:38.808930 28130 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:00:38.791777 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.22.208.129:35125 (local address 127.22.208.130:36403)
0812 08:00:38.793225 (+  1448us) server_negotiation.cc:207] Beginning negotiation
0812 08:00:38.793230 (+     5us) server_negotiation.cc:400] Waiting for connection header
0812 08:00:38.808633 (+ 15403us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.208.129:35125: BlockingRecv error: recv got EOF from 127.22.208.129:35125 (error 108)
Metrics: {"server-negotiator.queue_time_us":1344}
W20260812 08:00:38.809466 28778 negotiation.cc:336] Failed RPC negotiation. Trace:
0812 08:00:38.791412 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.22.208.129:34921 (local address 127.22.208.135:33345)
0812 08:00:38.795620 (+  4208us) server_negotiation.cc:207] Beginning negotiation
0812 08:00:38.795627 (+     7us) server_negotiation.cc:400] Waiting for connection header
0812 08:00:38.795650 (+    23us) server_negotiation.cc:408] Connection header received
0812 08:00:38.795731 (+    81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0812 08:00:38.795735 (+     4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0812 08:00:38.795942 (+   207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0812 08:00:38.796076 (+   134us) server_negotiation.cc:227] Negotiated authn=SASL
0812 08:00:38.808081 (+ 12005us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.208.129:34921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":4079,"thread_start_us":130,"threads_started":1}
I20260812 08:00:38.809842 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:40873
--local_ip_for_outbound_sockets=127.22.208.129
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37489
--webserver_interface=127.22.208.129
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:38.809988 24453 connection.cc:573] server connection from 127.22.208.129:56565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:38.814699 28768 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:38.819700 28009 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51690: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:38.824867 28363 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 3e1efe84be134f1bb72a09c4c17aa52f)
I20260812 08:00:38.825050 28363 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.825987 28363 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:38.826733 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:38.826960 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:38.827395 28071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:38.827576 28071 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:38.827672 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:38.827853 27473 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:38.835013 28635 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=2 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 08:00:38.835750 28635 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap complete.
W20260812 08:00:38.836007 28635 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 30ms of latency in SegmentAllocator::Sync()
I20260812 08:00:38.837919 28514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" is_pre_election: true
I20260812 08:00:38.838340 24453 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:38.838614 28363 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 17 FOLLOWER]: Advancing to term 18
I20260812 08:00:38.839916 28363 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260812 08:00:38.841391 24450 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Network error: Client connection negotiation failed: client connection to 127.22.208.129:40873: connect: Connection refused (error 111)
I20260812 08:00:38.840468 28707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
W20260812 08:00:38.842907 24453 leader_election.cc:343] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 18 pre-election: Tablet error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Illegal state: must be running to vote when last-logged opid is not known
W20260812 08:00:38.846246 28587 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:38.846334 28587 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 08:00:38.846366 28587 fault_injection.cc:59] injecting fault for kudu (pid 28379): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
W20260812 08:00:38.862221 24040 connection.cc:573] server connection from 127.22.208.134:52631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:38.866658 28635 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.116s	user 0.079s	sys 0.002s
I20260812 08:00:38.870522 28635 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.873713 28635 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:38.874312 28635 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:38.875705 28635 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260812 08:00:38.884680 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:38.885671 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20260812 08:00:38.898411 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:38.975391 28779 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:38.975759 28779 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:38.975808 28779 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:38.975903 28779 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:38.975944 28779 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:38.976053 28779 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:38.980952 28779 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:38.981060 28779 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:38.981079 28779 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:38.981140 28779 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:00:38.986379 28779 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:40873
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=37489
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.28779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:38.987598 28779 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:38.988807 28779 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:38.992717 28779 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:38.997450 28791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.997454 28792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:38.997453 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:38.998458 28779 server_base.cc:1061] running on GCE node
I20260812 08:00:38.998982 28779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:38.999791 28779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:39.000988 28779 hybrid_clock.cc:648] HybridClock initialized: now 1786521639000968 us; error 38 us; skew 500 ppm
I20260812 08:00:39.004067 28779 webserver.cc:533] Webserver started at http://127.22.208.129:37489/ using document root <none> and password file <none>
I20260812 08:00:39.004954 28779 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:39.005071 28779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:39.009421 28779 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:00:39.017959 28806 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:39.019589 28779 fs_manager.cc:730] Time spent opening block manager: real 0.009s	user 0.001s	sys 0.000s
I20260812 08:00:39.019912 28779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
uuid: "f29b76cef775415dbd2c0997fadb3904"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:39.020855 28779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:39.052134 28779 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:39.053151 28779 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:39.053406 28779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:39.054281 28779 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:39.055845 28813 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:39.058739 27421 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40776: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.060395 28779 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:39.060472 28779 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.001s	sys 0.000s
I20260812 08:00:39.060577 28779 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:39.062309 28779 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:39.062384 28779 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
I20260812 08:00:39.062902 28813 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap starting.
I20260812 08:00:39.075868 28779 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:40873
I20260812 08:00:39.077862 28779 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb
I20260812 08:00:39.078887 28944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:40873 every 8 connection(s)
I20260812 08:00:39.080157 28813 log.cc:826] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:39.087468 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 28779
I20260812 08:00:39.087710 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 27946
I20260812 08:00:39.102842 28945 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:39.103184 28945 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:39.103802 28945 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:39.106314 24051 ts_manager.cc:194] Re-registered known tserver with Master: f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873)
I20260812 08:00:39.107520 24051 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:33857
I20260812 08:00:39.108969 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:36403
--local_ip_for_outbound_sockets=127.22.208.130
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=33433
--webserver_interface=127.22.208.130
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:39.120429 28945 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:39.146314 28813 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=1 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 08:00:39.147076 28813 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Bootstrap complete.
W20260812 08:00:39.147265 28813 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 149ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.161593 28786 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:39.161831 28786 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.164058 28786 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:39.167073 28622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:39.176043 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:39.176276 26747 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
I20260812 08:00:39.177956 27473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:39.178050 28623 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.178154 27473 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e1efe84be134f1bb72a09c4c17aa52f in current term 18: Already voted for candidate b6d90cd6408d4dc3b47ebf57ca72e91b in this term.
W20260812 08:00:39.178313 28622 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:39.178614 28618 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:39.179610 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:39.179764 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 18.
I20260812 08:00:39.179260 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 18 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:39.181124 28786 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 17 FOLLOWER]: Advancing to term 18
W20260812 08:00:39.181282 28618 leader_election.cc:343] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 18 pre-election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:39.183341 28786 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260812 08:00:39.219998 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.262099 28950 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:39.262465 28950 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:39.262518 28950 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:39.262634 28950 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:39.262702 28950 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:39.262809 28950 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:39.267020 28950 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:39.267166 28950 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:39.267235 28950 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:39.267342 28950 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:00:39.272936 28950 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:36403
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=33433
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.28950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:39.274313 28950 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:39.275700 28950 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:39.277872 28964 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:39.278007 28964 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
E20260812 08:00:39.279179 28950 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260812 08:00:39.279372 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:39.279528 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 18.
I20260812 08:00:39.279677 28964 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:39.281236 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:39.281414 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 18.
W20260812 08:00:39.285418 27387 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
W20260812 08:00:39.287559 27388 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.287560 28707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:39.287776 28707 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 18.
W20260812 08:00:39.288189 28970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:39.288172 27388 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: ada36b19ad0e4386b68527e1ab0ce923, b6d90cd6408d4dc3b47ebf57ca72e91b
W20260812 08:00:39.288262 28974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:39.288393 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:39.288664 28964 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260812 08:00:39.288985 27384 leader_election.cc:343] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 pre-election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:39.289063 28950 server_base.cc:1061] running on GCE node
W20260812 08:00:39.289069 28971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:39.288767 28964 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:39.289403 28964 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 18 FOLLOWER]: Advancing to term 19
I20260812 08:00:39.289558 28950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:39.290294 28950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:39.290668 28964 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.291719 24528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:39.291891 24528 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 18 FOLLOWER]: Advancing to term 19
I20260812 08:00:39.291983 28964 leader_election.cc:290] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:39.292356 28707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
I20260812 08:00:39.292405 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:39.292487 28707 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 18 FOLLOWER]: Advancing to term 19
I20260812 08:00:39.292541 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 18 FOLLOWER]: Advancing to term 19
I20260812 08:00:39.292738 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "e3a0fdc906c74926946b47c61bd5cea5" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:39.293216 24528 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 19.
W20260812 08:00:39.293427 27384 leader_election.cc:343] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 election: Tablet error from VoteRequest() call to peer f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:39.293722 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 19.
I20260812 08:00:39.293748 28707 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a0fdc906c74926946b47c61bd5cea5 in term 19.
W20260812 08:00:39.294612 27387 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Network error: Client connection negotiation failed: client connection to 127.22.208.130:36403: connect: Connection refused (error 111)
I20260812 08:00:39.294790 27388 leader_election.cc:304] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, 9dede2273d3a4266a1bf7a5b368080f8, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5; no voters: ada36b19ad0e4386b68527e1ab0ce923, f29b76cef775415dbd2c0997fadb3904
W20260812 08:00:39.294946 27388 leader_election.cc:336] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.294945 28964 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Leader election won for term 19
I20260812 08:00:39.295291 28964 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 LEADER]: Becoming Leader. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Running, Role: LEADER
I20260812 08:00:39.295677 28964 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 19, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.296746 28813 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent bootstrapping tablet: real 0.234s	user 0.058s	sys 0.004s
I20260812 08:00:39.298097 28950 hybrid_clock.cc:648] HybridClock initialized: now 1786521639298059 us; error 47 us; skew 500 ppm
I20260812 08:00:39.299080 28813 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:39.300607 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 6ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.301182 28950 webserver.cc:533] Webserver started at http://127.22.208.130:33433/ using document root <none> and password file <none>
I20260812 08:00:39.302232 28950 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:39.302337 28950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:39.302449 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 reported cstate change: term changed from 18 to 19, leader changed from b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134) to e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132). New cstate: current_term: 19 leader_uuid: "e3a0fdc906c74926946b47c61bd5cea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: HEALTHY } } }
W20260812 08:00:39.303067 24476 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51550: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.303304 28813 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Initialized, Role: FOLLOWER
I20260812 08:00:39.303931 28813 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.305055 28813 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260812 08:00:39.306205 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:39.307313 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:39.310858 28950 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:00:39.320490 28997 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:39.322674 28950 fs_manager.cc:730] Time spent opening block manager: real 0.010s	user 0.005s	sys 0.000s
I20260812 08:00:39.322860 28950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
uuid: "ada36b19ad0e4386b68527e1ab0ce923"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:39.323534 28950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/data
Total live blocks: 7
Total live bytes: 7120
Total live bytes (after alignment): 32768
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:39.349326 28950 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:39.350069 28950 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:39.350260 28950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:39.351119 28950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:39.352908 29006 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:39.359134 28950 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:39.359243 28950 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
I20260812 08:00:39.359324 28950 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:39.362043 28950 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:39.362133 28950 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:00:39.362470 29006 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap starting.
I20260812 08:00:39.376631 28950 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:36403
I20260812 08:00:39.376684 29134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:36403 every 8 connection(s)
I20260812 08:00:39.377913 28950 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb
W20260812 08:00:39.380856 28964 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:39.380959 28964 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260812 08:00:39.381840 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 28950
I20260812 08:00:39.382045 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 24433
I20260812 08:00:39.382167 24528 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 19 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 19 index: 96. (index mismatch)
I20260812 08:00:39.398427 29006 log.cc:826] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:39.404410 28892 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 17 FOLLOWER]: Advancing to term 19
W20260812 08:00:39.405370 27387 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 -> Peer ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403): Couldn't send request to peer ada36b19ad0e4386b68527e1ab0ce923. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:39.408209 28892 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 19 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 19 index: 96. (index mismatch)
I20260812 08:00:39.409264 28980 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:39.411484 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.414618 26747 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 19 index: 96. (index mismatch)
I20260812 08:00:39.415306 28980 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:39.416939 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 17ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.419087 29135 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:39.419387 29135 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:39.420228 29135 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:39.422353 24050 ts_manager.cc:194] Re-registered known tserver with Master: ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403)
I20260812 08:00:39.423230 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:39533
--local_ip_for_outbound_sockets=127.22.208.131
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=43791
--webserver_interface=127.22.208.131
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:39.423276 27388 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 -> Peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533): Couldn't send request to peer 9dede2273d3a4266a1bf7a5b368080f8. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.131:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:00:39.424134 24050 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:60539
I20260812 08:00:39.427459 28707 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Refusing update from remote peer e3a0fdc906c74926946b47c61bd5cea5: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 19 index: 96. (index mismatch)
I20260812 08:00:39.428767 28980 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:39.429258 27388 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 -> Peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Couldn't send request to peer b6d90cd6408d4dc3b47ebf57ca72e91b. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:00:39.435225 29146 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 71ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.440973 29135 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:39.453307 29006 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 08:00:39.453928 29006 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Bootstrap complete.
I20260812 08:00:39.454372 29006 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent bootstrapping tablet: real 0.092s	user 0.058s	sys 0.011s
I20260812 08:00:39.458514 29006 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.461380 29006 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada36b19ad0e4386b68527e1ab0ce923, State: Initialized, Role: FOLLOWER
I20260812 08:00:39.461917 29006 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.462862 29006 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Time spent starting tablet: real 0.008s	user 0.011s	sys 0.001s
I20260812 08:00:39.471211 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:39.472482 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20260812 08:00:39.502805 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.513593 28984 mvcc.cc:204] Tried to move back new op lower bound from 7317592629064785920 to 7317592628630437888. Current Snapshot: MvccSnapshot[applied={T|T < 7317592624483827712}]
I20260812 08:00:39.518791 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.522524 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.525139 27391 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.006s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":162,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.525897 27527 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:39.526154 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.527793 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.528098 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.529573 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.530776 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.531073 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.532708 29174 mvcc.cc:204] Tried to move back new op lower bound from 7317592629064785920 to 7317592628630437888. Current Snapshot: MvccSnapshot[applied={T|T < 7317592628899028992 or (T in {7317592628976230400})}]
W20260812 08:00:39.532955 27899 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 129ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.535732 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.537314 26678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.537813 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.539162 26675 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:56354 (request call id 84) took 1508 ms. Trace:
W20260812 08:00:39.539285 26679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.539239 26675 rpcz_store.cc:276] 0812 08:00:38.030683 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:38.030744 (+    61us) service_pool.cc:224] Handling call
0812 08:00:38.031297 (+   553us) tablet_service.cc:2890] Created scanner 4d5c7c5c194947cea4a0bf85b2101b8d for tablet a12d50650402488683716909cc273d1e, query id is 19f7ff22fdb7433eaa325ec3ee25d231
0812 08:00:38.032464 (+  1167us) tablet_service.cc:3030] Creating iterator
0812 08:00:38.032542 (+    78us) tablet_service.cc:3408] Waiting safe time to advance
0812 08:00:38.422979 (+390437us) tablet_service.cc:3415] Waiting for operations to commit
0812 08:00:39.534221 (+1111242us) tablet_service.cc:3431] All operations in snapshot committed. Waited for 1501662 microseconds
0812 08:00:39.534404 (+   183us) tablet_service.cc:3055] Iterator created
0812 08:00:39.537283 (+  2879us) tablet_service.cc:3077] Iterator init: OK
0812 08:00:39.537349 (+    66us) tablet_service.cc:3120] has_more: true
0812 08:00:39.537430 (+    81us) tablet_service.cc:3137] Continuing scan request
0812 08:00:39.537532 (+   102us) tablet_service.cc:3201] Found scanner 4d5c7c5c194947cea4a0bf85b2101b8d for tablet a12d50650402488683716909cc273d1e, query id is 19f7ff22fdb7433eaa325ec3ee25d231
0812 08:00:39.539151 (+  1619us) inbound_call.cc:177] Queueing success response
Metrics: {"cfile_cache_hit":3,"cfile_cache_hit_bytes":686,"cfile_cache_miss":6,"cfile_cache_miss_bytes":1062,"cfile_init":2,"delta_iterators_relevant":1,"lbm_read_time_us":210,"lbm_reads_lt_1ms":14,"rowset_iterators":2,"scanner_bytes_read":439}
W20260812 08:00:39.539638 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 84ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.540087 25324 scanner-internal.cc:458] Time spent opening tablet: real 6.658s	user 0.006s	sys 0.000s
W20260812 08:00:39.544356 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:39.545593 29146 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 17ms of latency in SegmentAllocator::Sync()
E20260812 08:00:39.546782 29156 fault_injection.cc:59] injecting fault for kudu (pid 27356): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:39.552520 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.556869 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 51ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.561192 25208 connection.cc:573] client connection to 127.22.208.132:35833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:00:39.565326 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.034s	user 0.004s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":200,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.565845 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.565954 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.568882 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":126,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.569281 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.581815 29146 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 19ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.585130 29033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.586043 29030 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.586444 29033 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.586740 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.587301 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.592340 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.606386 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.037s	user 0.000s	sys 0.010s Metrics: {"bytes_written":4748,"cfile_init":1,"compiler_manager_pool.queue_time_us":241,"dirs.queue_time_us":5419,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1573,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":306,"threads_started":2}
I20260812 08:00:39.608287 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 212 bytes on disk
I20260812 08:00:39.610249 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4}
I20260812 08:00:39.611176 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.368508
I20260812 08:00:39.611378 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.059s	user 0.006s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":217,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.611948 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.614557 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:39.619262 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:39.623168 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.625351 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.099s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":143,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.626014 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.626929 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 98ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.627389 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 92ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.628219 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.629520 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.630750 29145 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:39.631063 29145 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:39.631096 29145 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:39.631150 29145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:39.631177 29145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:39.631251 29145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:39.640360 29145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:39.640622 29145 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:39.640715 29145 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:39.640890 29145 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
W20260812 08:00:39.645136 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.648132 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.648108 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.037s	user 0.016s	sys 0.004s Metrics: {"bytes_written":5827,"cfile_cache_hit":18,"cfile_cache_hit_bytes":2481,"cfile_cache_miss":30,"cfile_cache_miss_bytes":3381,"cfile_init":7,"delta_iterators_relevant":10,"dirs.queue_time_us":63,"drs_written":1,"lbm_read_time_us":1186,"lbm_reads_lt_1ms":58,"lbm_write_time_us":1431,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":45,"thread_start_us":543,"threads_started":5}
I20260812 08:00:39.649565 28769 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
W20260812 08:00:39.649724 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.650039 28633 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20260812 08:00:39.653062 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.653184 29145 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:39533
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=43791
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.29145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:39.654803 29145 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:39.657183 29145 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:39.657545 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
E20260812 08:00:39.661860 29145 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:39.662734 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.663934 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.668355 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.671627 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:39.675874 29201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:39.676151 29203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:39.676553 29200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:39.688822 29145 server_base.cc:1061] running on GCE node
I20260812 08:00:39.689386 29145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:39.690127 29145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:39.693878 29032 tablet_service.cc:3080] error setting up scanner c66bec5faf28437a947e91337c40e95e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.239s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
I20260812 08:00:39.696105 29145 hybrid_clock.cc:648] HybridClock initialized: now 1786521639695911 us; error 200 us; skew 500 ppm
I20260812 08:00:39.699213 29145 webserver.cc:533] Webserver started at http://127.22.208.131:43791/ using document root <none> and password file <none>
W20260812 08:00:39.699383 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.700139 29145 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:39.700269 29145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:39.706629 29145 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
W20260812 08:00:39.709254 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
W20260812 08:00:39.714258 28839 tablet_service.cc:3080] error setting up scanner 2069e4bb4fea4a138023aa6c73fad41b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.201s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
W20260812 08:00:39.715209 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.721689 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.095s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":113,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:39.722329 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:39.723333 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.727715 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.114s	user 0.010s	sys 0.003s Metrics: {"bytes_written":4748,"cfile_init":1,"compiler_manager_pool.queue_time_us":21057,"dirs.queue_time_us":5179,"dirs.run_cpu_time_us":444,"dirs.run_wall_time_us":1838,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":241,"threads_started":2}
W20260812 08:00:39.728382 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 82ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.728785 29216 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:39.728930 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.375235
I20260812 08:00:39.730167 29145 fs_manager.cc:730] Time spent opening block manager: real 0.022s	user 0.002s	sys 0.000s
I20260812 08:00:39.730307 29145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
uuid: "9dede2273d3a4266a1bf7a5b368080f8"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:39.731328 29145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:39.742262 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.743134 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.750792 26675 tablet_service.cc:3080] error setting up scanner 7591883fdda84041af9578c3e4c0ff13 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.239s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
W20260812 08:00:39.756774 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.763069 29145 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:39.764230 29145 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:39.764539 29145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:39.765580 29145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 08:00:39.765805 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.767379 29226 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:39.768723 28662 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:33602: Illegal state: replica 3e1efe84be134f1bb72a09c4c17aa52f is not leader of this config: current role FOLLOWER
I20260812 08:00:39.773181 29145 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:39.773280 29145 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.001s
I20260812 08:00:39.773365 29145 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:39.775472 29145 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:39.775558 29145 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.001s
W20260812 08:00:39.780682 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.783144 29226 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Bootstrap starting.
W20260812 08:00:39.787317 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.802039 29148 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:39.802258 29148 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.802441 29145 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:39533
I20260812 08:00:39.803898 29145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb
I20260812 08:00:39.804136 29148 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:39.810371 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 29145
I20260812 08:00:39.811132 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:35833
--local_ip_for_outbound_sockets=127.22.208.132
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37633
--webserver_interface=127.22.208.132
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:39.812008 29168 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:39.812152 29168 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.812840 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.090s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4748,"cfile_init":1,"dirs.queue_time_us":16327,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":2467,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":98,"threads_started":1}
I20260812 08:00:39.813025 29168 leader_election.cc:290] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:39.813791 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 212 bytes on disk
I20260812 08:00:39.813997 29185 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:39.814131 29185 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.814327 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20260812 08:00:39.814831 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.368508
I20260812 08:00:39.815303 29185 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:39.816357 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:39.816588 28892 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 19.
W20260812 08:00:39.817462 28998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:39.817510 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:39.817657 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 19.
I20260812 08:00:39.828950 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:39.829242 28892 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 19 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
W20260812 08:00:39.830155 28343 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 38ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.832221 29349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:39533 every 8 connection(s)
I20260812 08:00:39.833674 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:39.834045 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 18.
I20260812 08:00:39.834658 28622 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:39.834942 29185 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260812 08:00:39.835006 29185 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Starting leader election (detected failure of leader e3a0fdc906c74926946b47c61bd5cea5)
I20260812 08:00:39.835052 29185 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 19 FOLLOWER]: Advancing to term 20
I20260812 08:00:39.835608 28707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:39.836721 29185 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.837566 29185 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:39.838407 28707 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for earlier term 19. Current term is 20.
W20260812 08:00:39.838864 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.839679 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:39.839850 26747 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 19 FOLLOWER]: Advancing to term 20
W20260812 08:00:39.840476 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.840998 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:39.841125 26747 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 20.
I20260812 08:00:39.841147 29068 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 18 FOLLOWER]: Advancing to term 20
W20260812 08:00:39.841624 28618 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:39.841708 28998 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
W20260812 08:00:39.842566 28623 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
W20260812 08:00:39.843387 28618 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:39.844020 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 20.
W20260812 08:00:39.844053 28623 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.845319 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904"
I20260812 08:00:39.845489 28892 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 19 FOLLOWER]: Advancing to term 20
I20260812 08:00:39.850468 28892 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 20.
I20260812 08:00:39.851022 28618 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, f29b76cef775415dbd2c0997fadb3904; no voters: b6d90cd6408d4dc3b47ebf57ca72e91b, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:39.851298 29185 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Leader election won for term 20
I20260812 08:00:39.851384 29185 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 LEADER]: Becoming Leader. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Running, Role: LEADER
I20260812 08:00:39.851751 29185 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 19.96, Last appended by leader: 96, Current term: 20, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.854058 29226 log.cc:826] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Log is configured to *not* fsync() on all Append() calls
W20260812 08:00:39.854312 29146 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 103ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.856006 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:39.856560 24050 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f reported cstate change: term changed from 19 to 20, leader changed from e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132) to 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135). New cstate: current_term: 20 leader_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
I20260812 08:00:39.857975 29002 leader_election.cc:400] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for earlier term 19. Current term is 20.
I20260812 08:00:39.859622 29002 leader_election.cc:403] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260812 08:00:39.860226 29002 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.860929 29148 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for earlier term 19. Current term is 20.
W20260812 08:00:39.864257 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:39.866961 29032 tablet_service.cc:3080] error setting up scanner adf90792a0a04191935e5726a29138ef with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.412s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
I20260812 08:00:39.871528 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.057s	user 0.013s	sys 0.001s Metrics: {"bytes_written":5827,"cfile_cache_hit":26,"cfile_cache_hit_bytes":3804,"cfile_cache_miss":22,"cfile_cache_miss_bytes":2058,"cfile_init":5,"delta_iterators_relevant":10,"drs_written":1,"lbm_read_time_us":932,"lbm_reads_lt_1ms":42,"lbm_write_time_us":760,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":45,"thread_start_us":415,"threads_started":5}
I20260812 08:00:39.872113 26801 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
I20260812 08:00:39.872886 26665 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
I20260812 08:00:39.898528 28707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" is_pre_election: true
I20260812 08:00:39.912367 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:39.915542 29002 leader_election.cc:343] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: Tablet error from VoteRequest() call to peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:39.922263 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:39.922535 29068 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 20: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
I20260812 08:00:39.938853 26747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 19 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:39.939327 26747 raft_consensus.cc:2368] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for earlier term 19. Current term is 20.
I20260812 08:00:39.939932 29002 leader_election.cc:400] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 19 pre-election: Vote denied by peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369) with higher term. Message: Invalid argument: T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for earlier term 19. Current term is 20.
I20260812 08:00:39.947307 29226 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=1 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20260812 08:00:39.947929 29226 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Bootstrap complete.
W20260812 08:00:39.948135 29226 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 11ms of latency in SegmentAllocator::Sync()
I20260812 08:00:39.949512 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:39.950781 28892 raft_consensus.cc:2393] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dbc6ed4943d5493caaf8b5feba124db1 in current term 20: Already voted for candidate 3e1efe84be134f1bb72a09c4c17aa52f in this term.
W20260812 08:00:39.950958 29185 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:00:39.951022 29185 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260812 08:00:39.949927 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 15ms of latency in SegmentAllocator::Sync()
W20260812 08:00:39.953382 26661 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Network error: Client connection negotiation failed: client connection to 127.22.208.132:35833: connect: Connection refused (error 111)
I20260812 08:00:39.953500 26661 leader_election.cc:304] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dbc6ed4943d5493caaf8b5feba124db1; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:39.954305 29168 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260812 08:00:39.954993 26664 leader_election.cc:336] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:39.955247 29068 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Refusing update from remote peer 3e1efe84be134f1bb72a09c4c17aa52f: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 20 index: 97. (index mismatch)
I20260812 08:00:39.955447 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
W20260812 08:00:39.955869 28623 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f -> Peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Couldn't send request to peer b6d90cd6408d4dc3b47ebf57ca72e91b. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
E20260812 08:00:39.956115 29182 fault_injection.cc:59] injecting fault for kudu (pid 28590): FLAGS_fault_crash_on_leader_request_fraction (process will exit)
I20260812 08:00:39.961695 29226 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Time spent bootstrapping tablet: real 0.179s	user 0.064s	sys 0.008s
I20260812 08:00:39.962543 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "dbc6ed4943d5493caaf8b5feba124db1" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:39.964159 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" candidate_term: 20 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:39.964346 29226 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 19 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.967321 29350 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
W20260812 08:00:39.967308 26664 leader_election.cc:343] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [CANDIDATE]: Term 20 pre-election: Tablet error from VoteRequest() call to peer 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:39.967521 29350 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:39.967718 29226 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 19 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dede2273d3a4266a1bf7a5b368080f8, State: Initialized, Role: FOLLOWER
I20260812 08:00:39.968273 29350 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:39.968298 29226 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 17.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:39.969504 29226 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260812 08:00:39.969743 29299 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 19 FOLLOWER]: Advancing to term 20
I20260812 08:00:39.971438 24050 ts_manager.cc:194] Re-registered known tserver with Master: 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533)
I20260812 08:00:39.972025 29299 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1efe84be134f1bb72a09c4c17aa52f in term 20.
I20260812 08:00:39.972473 24050 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:32827
I20260812 08:00:39.974756 29350 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:39.975775 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
W20260812 08:00:39.977197 29032 tablet_service.cc:3080] error setting up scanner e033ff626ee84e66b0bdf599dd5758b5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.522s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
I20260812 08:00:39.976785 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260812 08:00:39.978456 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.243s	user 0.018s	sys 0.001s Metrics: {"bytes_written":5827,"cfile_cache_hit":17,"cfile_cache_hit_bytes":2448,"cfile_cache_miss":27,"cfile_cache_miss_bytes":3298,"cfile_init":7,"delta_iterators_relevant":8,"dirs.queue_time_us":16268,"dirs.run_cpu_time_us":589,"dirs.run_wall_time_us":5814,"drs_written":1,"lbm_read_time_us":914,"lbm_reads_lt_1ms":55,"lbm_write_time_us":13649,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":25,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":45,"spinlock_wait_cycles":4352,"thread_start_us":7007,"threads_started":3}
I20260812 08:00:39.979069 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
I20260812 08:00:39.979481 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20260812 08:00:39.984337 29221 connection.cc:573] server connection from 127.22.208.135:43633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:39.985617 25211 connection.cc:573] client connection to 127.22.208.135:33345 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:39.990895 24043 connection.cc:573] server connection from 127.22.208.135:52363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:00:39.991561 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.009653 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.015648 29032 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.015833 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.033725 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.036689 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.041497 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.044613 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.059769 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.068049 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.070854 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.073436 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.099694 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.104298 29375 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:40.104617 29375 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:40.104640 29375 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:40.104682 29375 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:40.104708 29375 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:40.104772 29375 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:40.108945 29375 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:40.109108 29375 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:40.109158 29375 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:40.109246 29375 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:00:40.115118 29375 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:35833
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=37633
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.29375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:40.116441 29375 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:40.117921 29375 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:40.121220 29375 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:40.126447 29408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.126703 29410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.126824 26675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:56354: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.127061 29413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:40.127983 29375 server_base.cc:1061] running on GCE node
I20260812 08:00:40.128783 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:40.129588 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:40.129964 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.130791 29375 hybrid_clock.cc:648] HybridClock initialized: now 1786521640130738 us; error 43 us; skew 500 ppm
I20260812 08:00:40.134087 29375 webserver.cc:533] Webserver started at http://127.22.208.132:37633/ using document root <none> and password file <none>
I20260812 08:00:40.135144 29375 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:40.135244 29375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260812 08:00:40.136623 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.139595 29375 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:00:40.147195 29424 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:40.148968 29375 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.003s	sys 0.000s
I20260812 08:00:40.149158 29375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
uuid: "e3a0fdc906c74926946b47c61bd5cea5"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:40.149747 29375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/data
Total live blocks: 8
Total live bytes: 7256
Total live bytes (after alignment): 36864
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:40.160547 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.166915 29375 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:40.167680 29375 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:40.167936 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:40.168766 29375 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260812 08:00:40.169148 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.170801 29432 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:40.175887 29375 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:40.176005 29375 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20260812 08:00:40.176151 29375 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:40.177802 29375 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:40.177889 29375 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
W20260812 08:00:40.177965 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.178444 29432 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap starting.
I20260812 08:00:40.192876 29375 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:35833
I20260812 08:00:40.192965 29555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:35833 every 8 connection(s)
I20260812 08:00:40.194128 29375 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb
I20260812 08:00:40.197149 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 29375
I20260812 08:00:40.197381 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 26639
W20260812 08:00:40.199921 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.204725 29432 log.cc:826] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Log is configured to *not* fsync() on all Append() calls
W20260812 08:00:40.209371 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.209738 29560 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:40.209860 29560 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.212096 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.133:35369
--local_ip_for_outbound_sockets=127.22.208.133
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=38847
--webserver_interface=127.22.208.133
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:40.213917 29560 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:40.217105 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:40.218448 29556 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:40.218765 29556 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:40.219431 29556 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:40.222033 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:40.222280 29299 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 20.
I20260812 08:00:40.223111 24054 ts_manager.cc:194] Re-registered known tserver with Master: e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:40.223426 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:40.223620 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 20.
I20260812 08:00:40.224103 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:47689
W20260812 08:00:40.224431 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.227855 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:40.228236 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369): Network error: Client connection negotiation failed: client connection to 127.22.208.133:35369: connect: Connection refused (error 111)
W20260812 08:00:40.228410 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:40.234326 29501 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
W20260812 08:00:40.237143 28807 leader_election.cc:343] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: Tablet error from VoteRequest() call to peer e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833): Illegal state: must be running to vote when last-logged opid is not known
I20260812 08:00:40.237330 28807 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, f29b76cef775415dbd2c0997fadb3904; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:40.238107 29560 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260812 08:00:40.238461 29556 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:40.244405 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.256691 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.265682 29432 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=1 applied=96 ignored=76} inserts{seen=5 ignored=0} mutations{seen=3 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:40.266124 29432 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Bootstrap complete.
W20260812 08:00:40.266341 29432 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 124ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.267694 28839 tablet_service.cc:3080] error setting up scanner 1fcde0be77e640f8adf74ef09edf8b01 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.755s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592635551629312 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "3ca623dfe54c4dcaa65de63f6e2ea463"
W20260812 08:00:40.325925 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.326699 25323 meta_cache.cc:1510] marking tablet server b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711) as failed
I20260812 08:00:40.327487 25323 meta_cache.cc:1510] marking tablet server dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369) as failed
W20260812 08:00:40.328625 28839 tablet_service.cc:3080] error setting up scanner dff388036d1549f6b01778a656c3366d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.816s (maximum allowed 0.150s): new_scan_request { tablet_id: "a12d50650402488683716909cc273d1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7317592624589402112 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\311\322\360\323\006\"3\n\005slave\022*\n 35ef33a70eab4846a37b4010d8e5a103\020\001\030\001 \001(\001" signature: "<redacted>""=\214\214\327\267\370\251\206[Z\0257N\030\353\363\307\357r\006\006\366\372\226#\261\013\344\0260\332[\3456\263t{\336\342\207\277\276\217c\301\277M6|\202\033h\354\'\n-b\232\270a\236>\310 " signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "c2ce3365c0774725a0aeb9c85ac44a1f"
W20260812 08:00:40.336989 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.353953 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.367278 29568 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:40.367527 29568 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:40.367545 29568 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:40.367586 29568 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:40.367607 29568 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:40.367663 29568 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:40.371670 29568 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:40.371776 29568 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:40.371795 29568 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:40.371851 29568 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.133
W20260812 08:00:40.373658 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.376490 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.377233 29568 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.133:35369
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
--webserver_interface=127.22.208.133
--webserver_port=38847
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.29568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.133

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:40.378564 29568 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:40.380170 29568 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:00:40.383893 29568 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:40.388619 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.388628 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.389061 29582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:40.390933 29432 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent bootstrapping tablet: real 0.213s	user 0.061s	sys 0.010s
I20260812 08:00:40.391273 29568 server_base.cc:1061] running on GCE node
I20260812 08:00:40.391759 29568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:40.392436 29568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260812 08:00:40.393023 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.393356 29432 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.393857 29432 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0fdc906c74926946b47c61bd5cea5, State: Initialized, Role: FOLLOWER
I20260812 08:00:40.394438 29432 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 19.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.395464 29432 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:00:40.395820 29568 hybrid_clock.cc:648] HybridClock initialized: now 1786521640395793 us; error 45 us; skew 500 ppm
I20260812 08:00:40.398183 29568 webserver.cc:533] Webserver started at http://127.22.208.133:38847/ using document root <none> and password file <none>
I20260812 08:00:40.398936 29568 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:40.399035 29568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:40.402164 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:40.403654 29568 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:00:40.403285 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:40.404289 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.409157 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.410691 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 46ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.412024 29598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:40.413170 29568 fs_manager.cc:730] Time spent opening block manager: real 0.008s	user 0.003s	sys 0.001s
I20260812 08:00:40.413331 29568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
uuid: "dbc6ed4943d5493caaf8b5feba124db1"
format_stamp: "Formatted at 2026-08-12 08:00:28 on dist-test-slave-dhph"
I20260812 08:00:40.413864 29568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/data
Total live blocks: 6
Total live bytes: 7197
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:00:40.414412 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.415665 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.433576 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.435276 29568 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:40.435976 29568 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:40.436180 29568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:40.436849 29568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:40.438359 29606 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:40.443234 29568 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:40.443332 29568 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
I20260812 08:00:40.443446 29568 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:40.445874 29568 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:40.445945 29568 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
I20260812 08:00:40.446461 29606 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap starting.
W20260812 08:00:40.451463 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.455744 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.459941 29568 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.133:35369
I20260812 08:00:40.460556 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.056s	user 0.003s	sys 0.003s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":180,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":269,"threads_started":1,"update_count":0,"wal-append.queue_time_us":555}
I20260812 08:00:40.461021 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.461285 29568 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb
I20260812 08:00:40.463240 29731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.133:35369 every 8 connection(s)
I20260812 08:00:40.467770 29606 log.cc:826] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Log is configured to *not* fsync() on all Append() calls
W20260812 08:00:40.471385 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.471982 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 29568
I20260812 08:00:40.472640 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.134:44711
--local_ip_for_outbound_sockets=127.22.208.134
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37749
--webserver_interface=127.22.208.134
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
W20260812 08:00:40.473744 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 90ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.478693 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.481035 29732 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:40.481300 29732 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:40.481985 29732 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:40.484037 24054 ts_manager.cc:194] Re-registered known tserver with Master: dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369)
I20260812 08:00:40.485011 24054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.133:47119
W20260812 08:00:40.493762 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.497853 29732 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:40.527143 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.530222 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.532559 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.532651 29606 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=3 applied=96 ignored=84} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 08:00:40.533293 29606 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Bootstrap complete.
W20260812 08:00:40.533488 29606 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 56ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.535499 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.536402 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.537501 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.539242 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.541285 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.542537 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.543341 28839 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.543672 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.544813 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.549574 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.550992 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.552709 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.554862 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.555310 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.556360 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.557126 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.561417 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.565862 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.105s	user 0.010s	sys 0.001s Metrics: {"bytes_written":4748,"cfile_init":1,"compiler_manager_pool.queue_time_us":4140,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1717,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1172,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":97,"threads_started":1}
I20260812 08:00:40.566992 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.368508
W20260812 08:00:40.585245 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.587990 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.588815 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.021s	user 0.017s	sys 0.000s Metrics: {"bytes_written":5827,"cfile_cache_hit":21,"cfile_cache_hit_bytes":2586,"cfile_cache_miss":27,"cfile_cache_miss_bytes":3276,"cfile_init":7,"delta_iterators_relevant":10,"dirs.queue_time_us":1925,"dirs.run_cpu_time_us":436,"dirs.run_wall_time_us":3617,"drs_written":1,"lbm_read_time_us":857,"lbm_reads_lt_1ms":55,"lbm_write_time_us":661,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":45,"thread_start_us":1078,"threads_started":3}
W20260812 08:00:40.589520 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.590344 29606 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent bootstrapping tablet: real 0.144s	user 0.072s	sys 0.000s
I20260812 08:00:40.593336 29606 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.594391 29606 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc6ed4943d5493caaf8b5feba124db1, State: Initialized, Role: FOLLOWER
W20260812 08:00:40.596536 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.589536 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
I20260812 08:00:40.601051 29606 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 19.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.603049 29606 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260812 08:00:40.604197 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20260812 08:00:40.608594 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
I20260812 08:00:40.609800 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20260812 08:00:40.613930 29250 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.614249 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.614530 29745 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:40.614688 29745 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:40.615267 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.617573 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.619552 29745 leader_election.cc:290] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
W20260812 08:00:40.619884 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.619931 29221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:40.621804 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.629618 29501 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 21 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:40.629987 29501 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 for term 21 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
W20260812 08:00:40.630165 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.631489 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.632190 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.632817 29560 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:40.632984 29560 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.633831 29560 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:40.636684 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:40.636874 29299 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 20.
I20260812 08:00:40.637991 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 21 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:40.638195 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dede2273d3a4266a1bf7a5b368080f8 in term 20.
I20260812 08:00:40.638262 29681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 21 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:40.638665 29681 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dede2273d3a4266a1bf7a5b368080f8 for term 21 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
W20260812 08:00:40.638916 29221 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:40.638964 29501 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:40.639199 29501 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 19.
W20260812 08:00:40.639938 29221 leader_election.cc:336] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:40.640041 29221 leader_election.cc:304] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, b6d90cd6408d4dc3b47ebf57ca72e91b, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5
I20260812 08:00:40.640491 29745 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260812 08:00:40.641907 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923" is_pre_election: true
I20260812 08:00:40.642077 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 20.
W20260812 08:00:40.643819 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.645283 28810 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:40.646340 29681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
I20260812 08:00:40.646646 29681 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 20.
I20260812 08:00:40.647320 29560 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260812 08:00:40.647388 29560 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:00:40.647428 29560 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 20 FOLLOWER]: Advancing to term 21
W20260812 08:00:40.647644 28840 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54636: Illegal state: replica f29b76cef775415dbd2c0997fadb3904 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.648574 29560 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.649184 29560 leader_election.cc:290] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 election: Requested vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), ada36b19ad0e4386b68527e1ab0ce923 (127.22.208.130:36403), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:40.649773 29501 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5"
I20260812 08:00:40.649935 29501 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 19 FOLLOWER]: Advancing to term 21
I20260812 08:00:40.650372 29068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "ada36b19ad0e4386b68527e1ab0ce923"
I20260812 08:00:40.650494 29068 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 20 FOLLOWER]: Advancing to term 21
I20260812 08:00:40.651302 29501 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 21.
I20260812 08:00:40.651614 29068 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 21.
I20260812 08:00:40.651856 29756 raft_consensus.cc:493] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:00:40.651959 29756 raft_consensus.cc:515] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:40.653034 29756 leader_election.cc:290] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 9dede2273d3a4266a1bf7a5b368080f8 (127.22.208.131:39533), 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345), f29b76cef775415dbd2c0997fadb3904 (127.22.208.129:40873), dbc6ed4943d5493caaf8b5feba124db1 (127.22.208.133:35369), b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711), e3a0fdc906c74926946b47c61bd5cea5 (127.22.208.132:35833)
I20260812 08:00:40.654723 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 22 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8" is_pre_election: true
I20260812 08:00:40.654886 29299 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada36b19ad0e4386b68527e1ab0ce923 in term 20.
W20260812 08:00:40.660140 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:40.660406 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:40.661002 29299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "9dede2273d3a4266a1bf7a5b368080f8"
I20260812 08:00:40.661185 29299 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 20 FOLLOWER]: Advancing to term 21
I20260812 08:00:40.662322 29681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "f29b76cef775415dbd2c0997fadb3904" candidate_term: 21 candidate_status { last_received { term: 19 index: 96 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1"
I20260812 08:00:40.662474 29681 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 20 FOLLOWER]: Advancing to term 21
I20260812 08:00:40.662791 29299 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 21.
I20260812 08:00:40.665038 29681 raft_consensus.cc:2468] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29b76cef775415dbd2c0997fadb3904 in term 21.
I20260812 08:00:40.665604 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 22 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
I20260812 08:00:40.665746 28810 leader_election.cc:304] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: ada36b19ad0e4386b68527e1ab0ce923, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904; no voters: 
I20260812 08:00:40.665794 28892 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 22 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
I20260812 08:00:40.666671 29560 raft_consensus.cc:2804] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 21 FOLLOWER]: Leader election won for term 21
I20260812 08:00:40.666941 29560 raft_consensus.cc:697] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [term 21 LEADER]: Becoming Leader. State: Replica: f29b76cef775415dbd2c0997fadb3904, State: Running, Role: LEADER
I20260812 08:00:40.667405 29560 consensus_queue.cc:237] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 19.96, Last appended by leader: 96, Current term: 21, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:40.671698 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 20ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.674471 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:40.674703 28810 leader_election.cc:336] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:40.675112 24054 catalog_manager.cc:5719] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 reported cstate change: term changed from 20 to 21, leader changed from 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135) to f29b76cef775415dbd2c0997fadb3904 (127.22.208.129). New cstate: current_term: 21 leader_uuid: "f29b76cef775415dbd2c0997fadb3904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } health_report { overall_health: UNKNOWN } } }
W20260812 08:00:40.678471 29613 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35918: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.681079 29613 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35918: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.682224 28892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "9dede2273d3a4266a1bf7a5b368080f8" candidate_term: 21 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "f29b76cef775415dbd2c0997fadb3904" is_pre_election: true
W20260812 08:00:40.684091 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.687153 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.690328 29501 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 22 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "e3a0fdc906c74926946b47c61bd5cea5" is_pre_election: true
I20260812 08:00:40.690554 29501 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 22 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
W20260812 08:00:40.691551 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.692589 29620 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35918: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.693246 29610 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35918: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.693228 29738 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:40.693545 29738 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:40.693576 29738 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:40.693661 29738 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:40.693702 29738 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:40.693799 29738 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:40.698452 29613 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35918: Illegal state: replica dbc6ed4943d5493caaf8b5feba124db1 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.699353 29681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a12d50650402488683716909cc273d1e" candidate_uuid: "ada36b19ad0e4386b68527e1ab0ce923" candidate_term: 22 candidate_status { last_received { term: 17 index: 95 } } ignore_live_leader: false dest_uuid: "dbc6ed4943d5493caaf8b5feba124db1" is_pre_election: true
W20260812 08:00:40.699460 29738 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:40.699585 29738 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:40.699635 29738 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:40.699728 29738 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.134
I20260812 08:00:40.699743 29681 raft_consensus.cc:2410] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada36b19ad0e4386b68527e1ab0ce923 for term 22 because replica has last-logged OpId of term: 19 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 95.
W20260812 08:00:40.702450 25211 meta_cache.cc:302] tablet a12d50650402488683716909cc273d1e: replica 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
W20260812 08:00:40.703769 29002 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111)
I20260812 08:00:40.703884 29002 leader_election.cc:304] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 9dede2273d3a4266a1bf7a5b368080f8, ada36b19ad0e4386b68527e1ab0ce923; no voters: 3e1efe84be134f1bb72a09c4c17aa52f, dbc6ed4943d5493caaf8b5feba124db1, e3a0fdc906c74926946b47c61bd5cea5, f29b76cef775415dbd2c0997fadb3904
I20260812 08:00:40.704094 29756 raft_consensus.cc:2749] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260812 08:00:40.704108 29002 leader_election.cc:336] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111)
I20260812 08:00:40.706985 29738 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.134:44711
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
--webserver_interface=127.22.208.134
--webserver_port=37749
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.29738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.134

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:40.708495 29738 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:40.710083 29738 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:40.711053 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
W20260812 08:00:40.713135 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
E20260812 08:00:40.714126 29738 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:40.716024 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 44ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.717038 29501 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5 [term 21 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 19 index: 96. Preceding OpId from leader: term: 21 index: 98. (index mismatch)
I20260812 08:00:40.717734 29560 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
W20260812 08:00:40.718895 28810 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 -> Peer b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711): Couldn't send request to peer b6d90cd6408d4dc3b47ebf57ca72e91b. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.134:44711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:00:40.720664 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 79ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.720817 29775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.721674 29773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:40.721835 29772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:40.722658 29738 server_base.cc:1061] running on GCE node
I20260812 08:00:40.723306 29738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:40.724133 29738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:40.728523 29068 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923 [term 21 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 21 index: 98. (index mismatch)
I20260812 08:00:40.728736 29681 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1 [term 21 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 19 index: 96. Preceding OpId from leader: term: 21 index: 98. (index mismatch)
I20260812 08:00:40.729444 29780 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 84, Time since last communication: 0.000s
I20260812 08:00:40.731170 29299 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8 [term 21 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 17 index: 95. Preceding OpId from leader: term: 21 index: 98. (index mismatch)
I20260812 08:00:40.731483 29780 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260812 08:00:40.731827 29779 consensus_queue.cc:1048] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 84, Time since last communication: 0.000s
I20260812 08:00:40.733934 29738 hybrid_clock.cc:648] HybridClock initialized: now 1786521640733827 us; error 122 us; skew 500 ppm
W20260812 08:00:40.739778 28810 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 -> Peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Couldn't send request to peer 3e1efe84be134f1bb72a09c4c17aa52f. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:00:40.742236 29457 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53950: Illegal state: replica e3a0fdc906c74926946b47c61bd5cea5 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.742767 29788 mvcc.cc:204] Tried to move back new op lower bound from 7317592629064785920 to 7317592628630437888. Current Snapshot: MvccSnapshot[applied={T|T < 7317592624483827712}]
W20260812 08:00:40.743971 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 78ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.744834 29738 webserver.cc:533] Webserver started at http://127.22.208.134:37749/ using document root <none> and password file <none>
I20260812 08:00:40.745636 29738 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:40.745714 29738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:40.749614 29782 mvcc.cc:204] Tried to move back new op lower bound from 7317592629064785920 to 7317592628630437888. Current Snapshot: MvccSnapshot[applied={T|T < 7317592628990238720 or (T in {7317592629031120896,7317592629001588736,7317592629064785920})}]
I20260812 08:00:40.751259 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.751556 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 158ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.756250 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.761103 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 121ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.761610 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 61ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.763384 29738 fs_manager.cc:714] Time spent opening directory manager: real 0.016s	user 0.003s	sys 0.000s
I20260812 08:00:40.774714 29812 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:40.775941 29738 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.003s	sys 0.000s
I20260812 08:00:40.776094 29738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:40.776592 29738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/data
Total live blocks: 6
Total live bytes: 6908
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:40.796327 29738 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:40.797286 29738 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:40.797509 29738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:40.798231 29738 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:40.799944 29819 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 08:00:40.807173 29738 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:40.807278 29738 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
I20260812 08:00:40.807337 29738 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:40.809306 29738 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:40.809389 29738 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:40.809739 29819 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap starting.
W20260812 08:00:40.828771 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 23ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.829331 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.073s	user 0.007s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":181,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.830471 29738 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.134:44711
I20260812 08:00:40.833176 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.833731 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.833726 29776 mvcc.cc:204] Tried to move back new op lower bound from 7317592640370814976 to 7317592640189337600. Current Snapshot: MvccSnapshot[applied={T|T < 7317592640370814976}]
I20260812 08:00:40.836971 29819 log.cc:826] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:40.840165 29796 mvcc.cc:204] Tried to move back new op lower bound from 7317592640370814976 to 7317592640189337600. Current Snapshot: MvccSnapshot[applied={T|T < 7317592640370814976}]
W20260812 08:00:40.840844 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 41ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.849272 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 29738
W20260812 08:00:40.849747 29252 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47254: Illegal state: replica 9dede2273d3a4266a1bf7a5b368080f8 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.849867 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.135:33345
--local_ip_for_outbound_sockets=127.22.208.135
--tserver_master_addrs=127.22.208.190:41383
--webserver_port=37235
--webserver_interface=127.22.208.135
--builtin_ntp_servers=127.22.208.148:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--raft_heartbeat_interval_ms=100
--log_preallocate_segments=false
--flush_threshold_mb=0
--flush_threshold_secs=1
--maintenance_manager_polling_interval_ms=10
--heartbeat_interval_ms=10
--update_tablet_stats_interval_ms=10 with env {}
I20260812 08:00:40.852370 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.852694 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 63ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.854539 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.862030 29031 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34424: Illegal state: replica ada36b19ad0e4386b68527e1ab0ce923 is not leader of this config: current role FOLLOWER
I20260812 08:00:40.867583 29738 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb
I20260812 08:00:40.869874 29944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.134:44711 every 8 connection(s)
W20260812 08:00:40.875785 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 23ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.882332 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 68ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.884467 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.890755 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.036s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":118,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.891182 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.896533 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 84ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.899520 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 31ms of latency in SegmentAllocator::Sync()
W20260812 08:00:40.912719 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 2ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.913081 29950 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:40.913384 29950 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:40.914134 29950 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
W20260812 08:00:40.916224 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 4ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.917778 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.080s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":152,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.918309 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.923764 24055 ts_manager.cc:194] Re-registered known tserver with Master: b6d90cd6408d4dc3b47ebf57ca72e91b (127.22.208.134:44711)
I20260812 08:00:40.924899 24055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.134:58523
I20260812 08:00:40.929340 29819 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap replayed 1/1 log segments. Stats: ops{read=104 overwritten=8 applied=84 ignored=75} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 12 replicates
I20260812 08:00:40.929944 29819 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Bootstrap complete.
I20260812 08:00:40.930390 29819 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent bootstrapping tablet: real 0.121s	user 0.069s	sys 0.010s
I20260812 08:00:40.933717 29819 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 FOLLOWER]: Replica starting. Triggering 12 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:40.934743 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 84ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.936614 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.051s	user 0.001s	sys 0.003s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.937335 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.937605 29819 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d90cd6408d4dc3b47ebf57ca72e91b, State: Initialized, Role: FOLLOWER
I20260812 08:00:40.938205 29819 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 18.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
W20260812 08:00:40.939419 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 3ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.939841 29950 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
I20260812 08:00:40.940750 29819 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260812 08:00:40.945734 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.194s	user 0.004s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.947782 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 656 bytes on disk
I20260812 08:00:40.949558 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.950609 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20260812 08:00:40.957227 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 18ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.958490 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.106s	user 0.006s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":209,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.959087 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:40.977288 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 99ms of latency in SegmentAllocator::Sync()
I20260812 08:00:40.979666 29890 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 18 FOLLOWER]: Advancing to term 21
I20260812 08:00:40.983561 29890 raft_consensus.cc:1275] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 21 FOLLOWER]: Refusing update from remote peer f29b76cef775415dbd2c0997fadb3904: Log matching property violated. Preceding OpId in replica: term: 18 index: 96. Preceding OpId from leader: term: 19 index: 96. (term mismatch)
I20260812 08:00:40.983685 29890 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting all ops after (but not including) 95
I20260812 08:00:40.983767 29890 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Aborting uncommitted NO_OP operation due to leader change: 18.96
I20260812 08:00:40.983855 29890 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:40.987757 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.029s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":125,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.987947 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.038s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.988233 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.988381 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:40.988722 29780 consensus_queue.cc:1050] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 84, Time since last communication: 0.000s
W20260812 08:00:40.993253 25324 scanner-internal.cc:458] Time spent opening tablet: real 1.302s	user 0.003s	sys 0.000s
I20260812 08:00:40.996570 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.105s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":126,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:40.997185 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.006361 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.088s	user 0.012s	sys 0.000s Metrics: {"bytes_written":196,"cfile_init":1,"compiler_manager_pool.queue_time_us":40039,"delete_count":4,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1529,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"reinsert_count":0,"rows_written":5,"thread_start_us":131,"threads_started":1,"update_count":0}
W20260812 08:00:41.012985 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 77ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.014175 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 212 bytes on disk
I20260812 08:00:41.014876 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.015767 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.016949 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.015s	user 0.002s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":171,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.017445 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.022338 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 1ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.023160 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 97ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.030295 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 29ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.038395 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.044301 29967 mvcc.cc:204] Tried to move back new op lower bound from 7317592641002995712 to 7317592634573303808. Current Snapshot: MvccSnapshot[applied={T|T < 7317592629001588736 or (T in {7317592629031120896})}]
I20260812 08:00:41.053043 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.035s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4748,"cfile_init":1,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":110,"threads_started":1}
I20260812 08:00:41.053725 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.063948 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 25ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.078372 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.089s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":106,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":1255808,"update_count":0}
I20260812 08:00:41.079026 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.091089 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.012s	user 0.010s	sys 0.000s Metrics: {"bytes_written":4748,"cfile_init":1,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"spinlock_wait_cycles":19712,"thread_start_us":94,"threads_started":1}
I20260812 08:00:41.091806 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.053s	user 0.004s	sys 0.002s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":214,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.092869 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.093573 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.094959 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.106s	user 0.011s	sys 0.000s Metrics: {"bytes_written":4797,"cfile_init":1,"compiler_manager_pool.queue_time_us":74512,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":774,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"rows_written":8,"thread_start_us":163,"threads_started":1}
I20260812 08:00:41.096092 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 246 bytes on disk
I20260812 08:00:41.098308 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.098964 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.100538 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 61ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.100980 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 108ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.106408 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.005s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":541,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.107049 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 212 bytes on disk
W20260812 08:00:41.109127 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 24ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.109560 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.094s	user 0.007s	sys 0.001s Metrics: {"bytes_written":4748,"cfile_init":1,"dirs.queue_time_us":50451,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":5,"thread_start_us":130,"threads_started":1}
I20260812 08:00:41.110569 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 212 bytes on disk
I20260812 08:00:41.112993 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.113834 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.115823 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.116523 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.116945 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 13ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.120814 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 66ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.123469 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.185s	user 0.003s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":176,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.124585 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.131697 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.016s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":142,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.128516 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.012s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4727,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2187,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":4}
I20260812 08:00:41.132548 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.132997 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 177ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.133273 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.410286
W20260812 08:00:41.133574 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 26ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.135525 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.082s	user 0.006s	sys 0.000s Metrics: {"bytes_written":4694,"cfile_init":1,"dirs.queue_time_us":29973,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":1592,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":503,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2,"thread_start_us":146,"threads_started":1}
I20260812 08:00:41.136328 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.147456 29949 flags.cc:432] Enabled unsafe flag: --log_inject_latency=true
W20260812 08:00:41.147800 29949 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_mean=30
W20260812 08:00:41.147835 29949 flags.cc:432] Enabled unsafe flag: --log_inject_latency_ms_stddev=60
W20260812 08:00:41.147905 29949 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:41.147938 29949 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:41.148031 29949 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:41.153636 29949 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:41.153765 29949 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260812 08:00:41.153800 29949 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260812 08:00:41.153882 29949 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.135
W20260812 08:00:41.159893 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 133ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.160588 29949 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_inject_latency=true
--log_inject_latency_ms_mean=30
--log_inject_latency_ms_stddev=60
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.135:33345
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
--webserver_interface=127.22.208.135
--webserver_port=37235
--flush_threshold_mb=0
--flush_threshold_secs=1
--heartbeat_interval_ms=10
--tserver_master_addrs=127.22.208.190:41383
--update_tablet_stats_interval_ms=10
--never_fsync=true
--heap_profile_path=/tmp/kudu.29949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.135

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:41.162030 29949 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260812 08:00:41.161931 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 93ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.163630 29949 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:41.164059 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.065s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":140,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.164774 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
E20260812 08:00:41.167405 29949 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:00:41.173131 29994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:41.173533 29992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:41.174096 29991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:41.174844 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.041s	user 0.015s	sys 0.002s Metrics: {"bytes_written":6057,"cfile_cache_hit":42,"cfile_cache_hit_bytes":4854,"cfile_cache_miss":32,"cfile_cache_miss_bytes":2712,"cfile_init":7,"delta_iterators_relevant":16,"dirs.queue_time_us":1477,"drs_written":1,"lbm_read_time_us":856,"lbm_reads_lt_1ms":60,"lbm_write_time_us":631,"lbm_writes_lt_1ms":26,"mutex_wait_us":19,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":54,"spinlock_wait_cycles":768,"thread_start_us":564,"threads_started":6}
I20260812 08:00:41.175688 29949 server_base.cc:1061] running on GCE node
I20260812 08:00:41.176308 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:41.177032 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:41.177438 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 844 bytes on disk
I20260812 08:00:41.177989 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.189323 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.060s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4781,"cfile_init":1,"dirs.queue_time_us":27123,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2354,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":790,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":7,"thread_start_us":94,"threads_started":1}
I20260812 08:00:41.192555 29949 hybrid_clock.cc:648] HybridClock initialized: now 1786521641192460 us; error 123 us; skew 500 ppm
I20260812 08:00:41.194733 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 223 bytes on disk
I20260812 08:00:41.200428 29949 webserver.cc:533] Webserver started at http://127.22.208.135:37235/ using document root <none> and password file <none>
I20260812 08:00:41.201195 29949 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:41.201265 29949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:41.203233 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.203951 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.206470 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.002s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":118,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.206905 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.211455 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.115s	user 0.001s	sys 0.001s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.212073 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.216814 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 108ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.219449 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.225140 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 57ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.225433 29949 fs_manager.cc:714] Time spent opening directory manager: real 0.022s	user 0.002s	sys 0.001s
W20260812 08:00:41.229000 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 86ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.257912 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.093s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4764,"cfile_init":1,"dirs.queue_time_us":16583,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":2437,"drs_written":1,"lbm_read_time_us":146,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1213,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":6,"thread_start_us":94,"threads_started":1}
I20260812 08:00:41.259162 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.640503
I20260812 08:00:41.262882 30018 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:41.268168 29949 fs_manager.cc:730] Time spent opening block manager: real 0.039s	user 0.003s	sys 0.001s
I20260812 08:00:41.268476 29949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
uuid: "3e1efe84be134f1bb72a09c4c17aa52f"
format_stamp: "Formatted at 2026-08-12 08:00:29 on dist-test-slave-dhph"
I20260812 08:00:41.269345 29949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/data
Total live blocks: 6
Total live bytes: 7197
Total live bytes (after alignment): 28672
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:41.284771 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.065s	user 0.003s	sys 0.003s Metrics: {"bytes_written":4663,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2111,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":673,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1}
I20260812 08:00:41.285552 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:41.294202 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.158s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":115,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.294780 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.299435 28810 consensus_peers.cc:597] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904 -> Peer 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345): Couldn't send request to peer 3e1efe84be134f1bb72a09c4c17aa52f. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.135:33345: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:00:41.302554 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 6ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.310392 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 74ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.308642 29949 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:41.311772 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.026s	user 0.015s	sys 0.000s Metrics: {"bytes_written":6074,"cfile_cache_hit":11,"cfile_cache_hit_bytes":1266,"cfile_cache_miss":29,"cfile_cache_miss_bytes":5446,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":284,"drs_written":1,"lbm_read_time_us":1683,"lbm_reads_lt_1ms":57,"lbm_write_time_us":673,"lbm_writes_lt_1ms":26,"mutex_wait_us":89,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":55}
I20260812 08:00:41.312326 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.312867 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.313838 29949 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:41.314046 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:41.314842 29949 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:41.317325 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.105s	user 0.012s	sys 0.001s Metrics: {"bytes_written":4977,"cfile_init":1,"compiler_manager_pool.queue_time_us":16461,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"rows_written":14,"thread_start_us":156,"threads_started":1}
I20260812 08:00:41.318461 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 302 bytes on disk
I20260812 08:00:41.324331 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.324972 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.327344 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.120s	user 0.004s	sys 0.002s Metrics: {"bytes_written":4694,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":2043,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2}
I20260812 08:00:41.328336 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 174 bytes on disk
I20260812 08:00:41.328936 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.330003 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.394646
I20260812 08:00:41.341432 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.046s	user 0.004s	sys 0.003s Metrics: {"bytes_written":4711,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":3}
I20260812 08:00:41.342344 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.465606
I20260812 08:00:41.342672 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.079s	user 0.015s	sys 0.008s Metrics: {"bytes_written":6057,"cfile_cache_hit":54,"cfile_cache_hit_bytes":4369,"cfile_cache_miss":20,"cfile_cache_miss_bytes":2989,"cfile_init":6,"delta_iterators_relevant":16,"dirs.queue_time_us":1437,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1104,"drs_written":1,"lbm_read_time_us":1127,"lbm_reads_lt_1ms":44,"lbm_write_time_us":743,"lbm_writes_lt_1ms":26,"mutex_wait_us":1351,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":54,"thread_start_us":328,"threads_started":5}
I20260812 08:00:41.343403 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 844 bytes on disk
I20260812 08:00:41.344062 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.344439 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.353499 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 31ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.357977 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.028s	user 0.011s	sys 0.005s Metrics: {"bytes_written":6057,"cfile_cache_hit":30,"cfile_cache_hit_bytes":3242,"cfile_cache_miss":40,"cfile_cache_miss_bytes":4184,"cfile_init":8,"delta_iterators_relevant":14,"dirs.queue_time_us":58,"drs_written":1,"lbm_read_time_us":1038,"lbm_reads_lt_1ms":72,"lbm_write_time_us":762,"lbm_writes_lt_1ms":26,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":54,"thread_start_us":359,"threads_started":6}
I20260812 08:00:41.358553 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 844 bytes on disk
I20260812 08:00:41.358970 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.359349 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.368024 30028 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260812 08:00:41.370430 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 61ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.371069 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.012s	user 0.005s	sys 0.001s Metrics: {"bytes_written":4663,"cfile_init":1,"dirs.queue_time_us":1127,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1}
I20260812 08:00:41.371645 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:41.375344 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.375875 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
W20260812 08:00:41.384923 29222 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260812 08:00:41.386878 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.254s	user 0.007s	sys 0.000s Metrics: {"bytes_written":4727,"cfile_init":1,"dirs.queue_time_us":634,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1931,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":4}
I20260812 08:00:41.387912 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.758973
I20260812 08:00:41.389727 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.064s	user 0.008s	sys 0.000s Metrics: {"bytes_written":4663,"cfile_init":1,"dirs.queue_time_us":28476,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":4225,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7376,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":1,"thread_start_us":103,"threads_started":1}
I20260812 08:00:41.390615 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:41.396867 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.397507 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.904979
W20260812 08:00:41.407423 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 129ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.422506 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 198ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.424379 29949 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 08:00:41.424458 29949 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.107s	user 0.001s	sys 0.000s
I20260812 08:00:41.424525 29949 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260812 08:00:41.426626 29949 ts_tablet_manager.cc:616] Registered 1 tablets
I20260812 08:00:41.426684 29949 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:41.435089 30028 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap starting.
I20260812 08:00:41.435521 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.091s	user 0.001s	sys 0.006s Metrics: {"bytes_written":4663,"cfile_init":1,"dirs.queue_time_us":91,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":3531,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1025,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1}
I20260812 08:00:41.436309 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 163 bytes on disk
I20260812 08:00:41.437668 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.438230 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:41.443379 29949 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.135:33345
I20260812 08:00:41.444558 29949 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb
I20260812 08:00:41.453967 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 29949
W20260812 08:00:41.454710 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 18ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.452527 30161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.135:33345 every 8 connection(s)
I20260812 08:00:41.458873 30028 log.cc:826] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:41.466674 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.064s	user 0.021s	sys 0.000s Metrics: {"bytes_written":6074,"cfile_cache_hit":24,"cfile_cache_hit_bytes":2808,"cfile_cache_miss":46,"cfile_cache_miss_bytes":4600,"cfile_init":10,"delta_iterators_relevant":14,"dirs.queue_time_us":18685,"dirs.run_cpu_time_us":1168,"dirs.run_wall_time_us":7311,"drs_written":1,"lbm_read_time_us":1345,"lbm_reads_lt_1ms":86,"lbm_write_time_us":7737,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":25,"mutex_wait_us":4188,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":55,"thread_start_us":265,"threads_started":5}
I20260812 08:00:41.467343 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.472644 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.475560 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.037s	user 0.009s	sys 0.002s Metrics: {"bytes_written":6074,"cfile_cache_hit":24,"cfile_cache_hit_bytes":4731,"cfile_cache_miss":16,"cfile_cache_miss_bytes":1981,"cfile_init":3,"delta_iterators_relevant":6,"drs_written":1,"lbm_read_time_us":535,"lbm_reads_lt_1ms":28,"lbm_write_time_us":589,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":55}
I20260812 08:00:41.477038 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.479542 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.494086 30162 heartbeater.cc:344] Connected to a master server at 127.22.208.190:41383
I20260812 08:00:41.494412 30162 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:41.495208 30162 heartbeater.cc:507] Master 127.22.208.190:41383 requested a full tablet report, sending...
I20260812 08:00:41.497594 24051 ts_manager.cc:194] Re-registered known tserver with Master: 3e1efe84be134f1bb72a09c4c17aa52f (127.22.208.135:33345)
I20260812 08:00:41.498605 24051 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.135:34557
I20260812 08:00:41.507134 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.119s	user 0.023s	sys 0.002s Metrics: {"bytes_written":6057,"cfile_cache_hit":35,"cfile_cache_hit_bytes":3157,"cfile_cache_miss":61,"cfile_cache_miss_bytes":4959,"cfile_init":13,"delta_iterators_relevant":20,"dirs.queue_time_us":48,"drs_written":1,"lbm_read_time_us":1897,"lbm_reads_lt_1ms":113,"lbm_write_time_us":681,"lbm_writes_lt_1ms":26,"num_input_rowsets":4,"peak_mem_usage":131072,"rows_written":54,"thread_start_us":366,"threads_started":6}
I20260812 08:00:41.508872 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 844 bytes on disk
I20260812 08:00:41.509595 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.511225 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.511226 30162 heartbeater.cc:499] Master 127.22.208.190:41383 was elected leader, sending a full tablet report...
W20260812 08:00:41.522068 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 106ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.540568 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.165s	user 0.009s	sys 0.005s Metrics: {"bytes_written":6074,"cfile_cache_hit":11,"cfile_cache_hit_bytes":1266,"cfile_cache_miss":29,"cfile_cache_miss_bytes":5446,"cfile_init":6,"delta_iterators_relevant":6,"dirs.queue_time_us":221,"drs_written":1,"lbm_read_time_us":10245,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":51,"lbm_write_time_us":685,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":55}
I20260812 08:00:41.541281 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.541949 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.544792 30028 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=2 applied=96 ignored=84} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 08:00:41.545732 30028 tablet_bootstrap.cc:492] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Bootstrap complete.
I20260812 08:00:41.546188 30028 ts_tablet_manager.cc:1403] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent bootstrapping tablet: real 0.111s	user 0.068s	sys 0.005s
I20260812 08:00:41.550012 30028 raft_consensus.cc:359] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:41.550765 30028 raft_consensus.cc:740] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1efe84be134f1bb72a09c4c17aa52f, State: Initialized, Role: FOLLOWER
I20260812 08:00:41.551383 30028 consensus_queue.cc:260] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 20.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dede2273d3a4266a1bf7a5b368080f8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39533 } } peers { permanent_uuid: "3e1efe84be134f1bb72a09c4c17aa52f" member_type: VOTER last_known_addr { host: "127.22.208.135" port: 33345 } } peers { permanent_uuid: "ada36b19ad0e4386b68527e1ab0ce923" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 36403 } } peers { permanent_uuid: "f29b76cef775415dbd2c0997fadb3904" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 40873 } } peers { permanent_uuid: "dbc6ed4943d5493caaf8b5feba124db1" member_type: VOTER last_known_addr { host: "127.22.208.133" port: 35369 } } peers { permanent_uuid: "b6d90cd6408d4dc3b47ebf57ca72e91b" member_type: VOTER last_known_addr { host: "127.22.208.134" port: 44711 } } peers { permanent_uuid: "e3a0fdc906c74926946b47c61bd5cea5" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 35833 } }
I20260812 08:00:41.552428 30028 ts_tablet_manager.cc:1434] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260812 08:00:41.561211 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 726 bytes on disk
I20260812 08:00:41.562379 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20260812 08:00:41.619352 30099 raft_consensus.cc:3060] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 20 FOLLOWER]: Advancing to term 21
I20260812 08:00:41.622262 30099 pending_rounds.cc:85] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting all ops after (but not including) 96
I20260812 08:00:41.622377 30099 pending_rounds.cc:107] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Aborting uncommitted NO_OP operation due to leader change: 20.97
I20260812 08:00:41.622958 30099 raft_consensus.cc:2889] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260812 08:00:41.630136 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.118s	user 0.005s	sys 0.000s Metrics: {"bytes_written":4663,"cfile_init":1,"dirs.queue_time_us":2992,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1638,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1}
I20260812 08:00:41.630997 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=0.014997
I20260812 08:00:41.645447 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.303s	user 0.015s	sys 0.003s Metrics: {"bytes_written":6074,"cfile_cache_hit":39,"cfile_cache_hit_bytes":3497,"cfile_cache_miss":53,"cfile_cache_miss_bytes":4755,"cfile_init":13,"delta_iterators_relevant":18,"dirs.queue_time_us":20943,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1038,"drs_written":1,"lbm_read_time_us":6905,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":104,"lbm_write_time_us":714,"lbm_writes_lt_1ms":26,"mutex_wait_us":20793,"num_input_rowsets":4,"peak_mem_usage":131072,"rows_written":55,"thread_start_us":390,"threads_started":6}
I20260812 08:00:41.646353 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.646983 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20260812 08:00:41.695685 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 137ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.701510 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.707484 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 63ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.773551 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.142s	user 0.007s	sys 0.008s Metrics: {"bytes_written":6074,"cfile_cache_hit":11,"cfile_cache_hit_bytes":1266,"cfile_cache_miss":29,"cfile_cache_miss_bytes":5446,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":317,"drs_written":1,"lbm_read_time_us":1013,"lbm_reads_lt_1ms":57,"lbm_write_time_us":811,"lbm_writes_lt_1ms":26,"mutex_wait_us":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":55,"spinlock_wait_cycles":3840}
I20260812 08:00:41.774183 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:41.774966 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20260812 08:00:41.834146 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 96ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.895593 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 21ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.899185 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.899781 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 133ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.900732 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 83ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.901538 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 141ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.908749 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 46ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.917201 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 39ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.931780 30026 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.203s	user 0.002s	sys 0.000s
I20260812 08:00:41.934295 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.233s	user 0.015s	sys 0.000s Metrics: {"bytes_written":4694,"cfile_init":1,"compiler_manager_pool.queue_time_us":7867,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":3261,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2,"spinlock_wait_cycles":710400,"thread_start_us":131,"threads_started":1}
I20260812 08:00:41.935619 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 174 bytes on disk
I20260812 08:00:41.936350 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20260812 08:00:41.937268 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 163ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.939924 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.940419 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.952454 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.953948 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 33ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.956763 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 22ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.959910 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.961395 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.979415 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 137ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.980465 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.040s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":115,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.980942 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.987423 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 26ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.988770 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.036s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":147,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.989290 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:41.996065 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:41.996790 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 20ms of latency in SegmentAllocator::Sync()
W20260812 08:00:41.997515 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 66ms of latency in SegmentAllocator::Sync()
I20260812 08:00:41.998066 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.038s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":109,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:41.998495 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.013850 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 62ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.019153 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.020s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":83,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:42.020602 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 75ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.021292 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
W20260812 08:00:42.036686 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 40ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.044651 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.047206 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 1ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.057979 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.060740 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.001s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":147,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.061111 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:42.065265 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":631,"cfile_cache_miss":2,"cfile_cache_miss_bytes":102,"cfile_init":1,"delete_count":49,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":53,"lbm_reads_lt_1ms":6,"lbm_write_time_us":91,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":119,"threads_started":1,"update_count":0}
I20260812 08:00:42.065336 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.069s	user 0.001s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":126,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.065804 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:42.066437 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 75ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.076515 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 30ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.077255 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 100ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.077780 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.088s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":101,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.078302 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.097939 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.076s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":6,"cfile_cache_hit_bytes":769,"delete_count":47,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"lbm_write_time_us":128,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":80,"threads_started":1,"update_count":0}
I20260812 08:00:42.098446 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.099918 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 35ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.100677 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 73ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.107721 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.109596 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 178ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.109613 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.031s	user 0.003s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":129,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.110014 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.004000
W20260812 08:00:42.110481 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 84ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.117374 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 8ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.127235 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.146s	user 0.001s	sys 0.001s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":106,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.128448 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.131469 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 100ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.137125 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.039s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":103,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:42.137987 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 50ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.138243 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.142102 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 4ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.146646 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 69ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.148020 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.082s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":4,"cfile_cache_hit_bytes":722,"delete_count":48,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_write_time_us":169,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":101,"threads_started":1,"update_count":0}
I20260812 08:00:42.148451 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.174135 30026 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.233s	user 0.000s	sys 0.001s
W20260812 08:00:42.174381 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 34ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.175709 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.236s	user 0.001s	sys 0.001s Metrics: {"bytes_written":223,"delete_count":6,"lbm_write_time_us":422,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.176249 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.178921 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.134s	user 0.001s	sys 0.001s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":196,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.179481 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.181377 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 16ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.188560 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 88ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.194887 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 103ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.196358 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.088s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":134,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.197129 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:42.199788 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.020s	user 0.000s	sys 0.001s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":242,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.200235 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.201417 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 42ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.211707 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.035s	user 0.002s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":97,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.212323 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling FlushMRSOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.218555 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.070s	user 0.001s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":94,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.219041 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:42.226061 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 164ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.226958 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.001s	sys 0.002s Metrics: {"cfile_cache_hit":4,"cfile_cache_hit_bytes":769,"delete_count":52,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_write_time_us":122,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:42.227495 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 140ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.228097 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.232295 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 83ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.233287 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.104s	user 0.000s	sys 0.001s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":191,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.233861 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.244277 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 133ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.245523 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.045s	user 0.000s	sys 0.001s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":99,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.246026 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.278584 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.140s	user 0.001s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":116,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.279131 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.281845 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":215,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.282346 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.004000
I20260812 08:00:42.285908 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.001s	sys 0.001s Metrics: {"cfile_cache_hit":3,"cfile_cache_hit_bytes":688,"cfile_cache_miss":5,"cfile_cache_miss_bytes":218,"cfile_init":2,"delete_count":54,"delta_blocks_compacted":4,"delta_iterators_relevant":4,"dirs.queue_time_us":76,"lbm_read_time_us":104,"lbm_reads_lt_1ms":13,"lbm_write_time_us":92,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:42.288168 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 24ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.298380 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 91ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.312899 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 77ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.314544 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.204s	user 0.005s	sys 0.000s Metrics: {"cfile_cache_hit":6,"cfile_cache_hit_bytes":769,"cfile_cache_miss":2,"cfile_cache_miss_bytes":69,"delete_count":48,"delta_blocks_compacted":4,"delta_iterators_relevant":4,"dirs.queue_time_us":241,"lbm_read_time_us":764,"lbm_reads_lt_1ms":2,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":81,"threads_started":1,"update_count":0}
I20260812 08:00:42.315322 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.317391 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.083s	user 0.002s	sys 0.000s Metrics: {"bytes_written":185,"delete_count":3,"lbm_write_time_us":140,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.317874 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.005000
W20260812 08:00:42.320914 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 66ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.367877 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 49ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.377663 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 1ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.379824 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.380175 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.134s	user 0.000s	sys 0.002s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":191,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.381371 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.382566 29783 log.cc:510] T a12d50650402488683716909cc273d1e P ada36b19ad0e4386b68527e1ab0ce923: Injecting 59ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.388031 29597 log.cc:510] T a12d50650402488683716909cc273d1e P e3a0fdc906c74926946b47c61bd5cea5: Injecting 74ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.389353 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.071s	user 0.001s	sys 0.004s Metrics: {"cfile_cache_hit":6,"cfile_cache_hit_bytes":769,"cfile_cache_miss":4,"cfile_cache_miss_bytes":193,"cfile_init":2,"delete_count":54,"delta_blocks_compacted":5,"delta_iterators_relevant":5,"lbm_read_time_us":143,"lbm_reads_lt_1ms":12,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":86,"threads_started":1,"update_count":0}
W20260812 08:00:42.390118 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 84ms of latency in SegmentAllocator::Sync()
W20260812 08:00:42.390748 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 23ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.391179 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.391840 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.195s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":684,"cfile_cache_miss":2,"cfile_cache_miss_bytes":69,"cfile_init":1,"delete_count":50,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":111,"lbm_reads_lt_1ms":6,"lbm_write_time_us":158,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.392123 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: FlushMRSOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.180s	user 0.008s	sys 0.000s Metrics: {"bytes_written":174,"cfile_init":1,"delete_count":2,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6526,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"reinsert_count":0,"rows_written":8,"update_count":0}
W20260812 08:00:42.392458 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 74ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.392596 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.392875 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling CompactRowSetsOp(a12d50650402488683716909cc273d1e): perf score=1.190834
W20260812 08:00:42.410238 30194 log.cc:510] T a12d50650402488683716909cc273d1e P 3e1efe84be134f1bb72a09c4c17aa52f: Injecting 102ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.415450 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.100s	user 0.002s	sys 0.000s Metrics: {"bytes_written":223,"delete_count":6,"lbm_write_time_us":115,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.416023 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
W20260812 08:00:42.417456 29789 log.cc:510] T a12d50650402488683716909cc273d1e P 9dede2273d3a4266a1bf7a5b368080f8: Injecting 30ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.418736 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.190s	user 0.003s	sys 0.000s Metrics: {"bytes_written":174,"delete_count":2,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.419355 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.422590 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.423089 29733 maintenance_manager.cc:419] P dbc6ed4943d5493caaf8b5feba124db1: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
W20260812 08:00:42.425742 29797 log.cc:510] T a12d50650402488683716909cc273d1e P dbc6ed4943d5493caaf8b5feba124db1: Injecting 64ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.443781 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.062s	user 0.002s	sys 0.001s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":92,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.444512 29136 maintenance_manager.cc:419] P ada36b19ad0e4386b68527e1ab0ce923: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.006000
I20260812 08:00:42.448951 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.033s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":111,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.449409 29351 maintenance_manager.cc:419] P 9dede2273d3a4266a1bf7a5b368080f8: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.003000
I20260812 08:00:42.451149 29003 maintenance_manager.cc:643] P ada36b19ad0e4386b68527e1ab0ce923: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.006s	user 0.006s	sys 0.000s Metrics: {"cfile_cache_hit":9,"cfile_cache_hit_bytes":915,"cfile_cache_miss":3,"cfile_cache_miss_bytes":116,"cfile_init":1,"delete_count":55,"delta_blocks_compacted":6,"delta_iterators_relevant":6,"dirs.queue_time_us":206,"lbm_read_time_us":216,"lbm_reads_lt_1ms":7,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":105,"threads_started":1,"update_count":0}
I20260812 08:00:42.453392 29222 maintenance_manager.cc:643] P 9dede2273d3a4266a1bf7a5b368080f8: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.004s	user 0.002s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":667,"cfile_cache_miss":4,"cfile_cache_miss_bytes":198,"cfile_init":2,"delete_count":55,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":74,"lbm_read_time_us":190,"lbm_reads_lt_1ms":12,"lbm_write_time_us":117,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.464176 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.077s	user 0.001s	sys 0.002s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.464919 29557 maintenance_manager.cc:419] P e3a0fdc906c74926946b47c61bd5cea5: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:42.468632 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.077s	user 0.001s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":96,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.469528 28947 maintenance_manager.cc:419] P f29b76cef775415dbd2c0997fadb3904: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:42.470268 29429 maintenance_manager.cc:643] P e3a0fdc906c74926946b47c61bd5cea5: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.005s	user 0.002s	sys 0.002s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":749,"cfile_cache_miss":2,"cfile_cache_miss_bytes":69,"cfile_init":1,"delete_count":55,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":84,"lbm_reads_lt_1ms":6,"lbm_write_time_us":283,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:42.471951 29143 log.cc:510] T a12d50650402488683716909cc273d1e P f29b76cef775415dbd2c0997fadb3904: Injecting 59ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.475831 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.083s	user 0.002s	sys 0.000s Metrics: {"bytes_written":196,"delete_count":4,"lbm_write_time_us":122,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.476351 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
W20260812 08:00:42.479995 29977 log.cc:510] T a12d50650402488683716909cc273d1e P b6d90cd6408d4dc3b47ebf57ca72e91b: Injecting 22ms of latency in SegmentAllocator::Sync()
I20260812 08:00:42.491957 29603 maintenance_manager.cc:643] P dbc6ed4943d5493caaf8b5feba124db1: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.069s	user 0.004s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":720,"cfile_cache_miss":4,"cfile_cache_miss_bytes":149,"cfile_init":2,"delete_count":55,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":61,"lbm_read_time_us":169,"lbm_reads_lt_1ms":12,"lbm_write_time_us":141,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.503887 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.027s	user 0.002s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":696,"cfile_cache_miss":2,"cfile_cache_miss_bytes":102,"cfile_init":1,"delete_count":54,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":89,"lbm_read_time_us":69,"lbm_reads_lt_1ms":6,"lbm_write_time_us":123,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.504343 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e): perf score=1.000000
I20260812 08:00:42.507320 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: FlushDeltaMemStoresOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.002s	sys 0.000s Metrics: {"bytes_written":163,"delete_count":1,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.507726 29951 maintenance_manager.cc:419] P b6d90cd6408d4dc3b47ebf57ca72e91b: Scheduling MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e): perf score=0.002000
I20260812 08:00:42.511157 29817 maintenance_manager.cc:643] P b6d90cd6408d4dc3b47ebf57ca72e91b: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.003s	user 0.002s	sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":818,"cfile_init":2,"delete_count":55,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":76,"lbm_read_time_us":110,"lbm_reads_lt_1ms":12,"lbm_write_time_us":123,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
I20260812 08:00:42.516032 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: CompactRowSetsOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.122s	user 0.019s	sys 0.000s Metrics: {"bytes_written":6074,"cfile_cache_hit":33,"cfile_cache_hit_bytes":3499,"cfile_cache_miss":41,"cfile_cache_miss_bytes":4311,"cfile_init":10,"delta_iterators_relevant":16,"dirs.queue_time_us":1414,"drs_written":1,"lbm_read_time_us":1178,"lbm_reads_lt_1ms":81,"lbm_write_time_us":578,"lbm_writes_lt_1ms":26,"mutex_wait_us":37,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":55,"spinlock_wait_cycles":12032,"thread_start_us":506,"threads_started":7}
I20260812 08:00:42.516758 30163 maintenance_manager.cc:419] P 3e1efe84be134f1bb72a09c4c17aa52f: Scheduling UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e): 856 bytes on disk
I20260812 08:00:42.517218 30026 maintenance_manager.cc:643] P 3e1efe84be134f1bb72a09c4c17aa52f: UndoDeltaBlockGCOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260812 08:00:42.532749 28811 maintenance_manager.cc:643] P f29b76cef775415dbd2c0997fadb3904: MinorDeltaCompactionOp(a12d50650402488683716909cc273d1e) complete. Timing: real 0.063s	user 0.002s	sys 0.000s Metrics: {"cfile_cache_hit":2,"cfile_cache_hit_bytes":749,"cfile_cache_miss":2,"cfile_cache_miss_bytes":69,"cfile_init":1,"delete_count":55,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":93,"lbm_read_time_us":105,"lbm_reads_lt_1ms":6,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":0}
W20260812 08:00:43.662066 25323 scanner-internal.cc:458] Time spent opening tablet: real 6.646s	user 0.000s	sys 0.006s
I20260812 08:00:43.734313 29266 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:43.736083 29056 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:43.744977 29859 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:43.749367 28864 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260812 08:00:43.750424 29480 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:43.755801 30091 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:00:43.756556 29656 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 10db3a93a2c74676a9fc532fdd526733 | 127.22.208.190:41383 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                                Value                                                                                                 |      Tags       |         Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                                                  | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                                                 | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                    | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                                                 | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                                 | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                                   | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                                                  | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:44797 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 3e1efe84be134f1bb72a09c4c17aa52f | 127.22.208.135:33345 | HEALTHY | <none>   |       0        |       0
 9dede2273d3a4266a1bf7a5b368080f8 | 127.22.208.131:39533 | HEALTHY | <none>   |       0        |       0
 ada36b19ad0e4386b68527e1ab0ce923 | 127.22.208.130:36403 | HEALTHY | <none>   |       0        |       0
 b6d90cd6408d4dc3b47ebf57ca72e91b | 127.22.208.134:44711 | HEALTHY | <none>   |       0        |       0
 dbc6ed4943d5493caaf8b5feba124db1 | 127.22.208.133:35369 | HEALTHY | <none>   |       0        |       0
 e3a0fdc906c74926946b47c61bd5cea5 | 127.22.208.132:35833 | HEALTHY | <none>   |       0        |       0
 f29b76cef775415dbd2c0997fadb3904 | 127.22.208.129:40873 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                  Flag                   |                                                                                              Value                                                                                               |         Tags         |      Tablet Server
-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
 flush_threshold_mb                      | 0                                                                                                                                                                                                | runtime,experimental | all 7 server(s) checked
 flush_threshold_secs                    | 1                                                                                                                                                                                                | runtime,experimental | all 7 server(s) checked
 ipki_server_key_size                    | 768                                                                                                                                                                                              | experimental         | all 7 server(s) checked
 local_ip_for_outbound_sockets           | 127.22.208.129                                                                                                                                                                                   | experimental         | 127.22.208.129:40873
 local_ip_for_outbound_sockets           | 127.22.208.130                                                                                                                                                                                   | experimental         | 127.22.208.130:36403
 local_ip_for_outbound_sockets           | 127.22.208.131                                                                                                                                                                                   | experimental         | 127.22.208.131:39533
 local_ip_for_outbound_sockets           | 127.22.208.132                                                                                                                                                                                   | experimental         | 127.22.208.132:35833
 local_ip_for_outbound_sockets           | 127.22.208.133                                                                                                                                                                                   | experimental         | 127.22.208.133:35369
 local_ip_for_outbound_sockets           | 127.22.208.134                                                                                                                                                                                   | experimental         | 127.22.208.134:44711
 local_ip_for_outbound_sockets           | 127.22.208.135                                                                                                                                                                                   | experimental         | 127.22.208.135:33345
 log_inject_latency                      | true                                                                                                                                                                                             | runtime,unsafe       | all 7 server(s) checked
 log_inject_latency_ms_mean              | 30                                                                                                                                                                                               | runtime,unsafe       | all 7 server(s) checked
 log_inject_latency_ms_stddev            | 60                                                                                                                                                                                               | runtime,unsafe       | all 7 server(s) checked
 maintenance_manager_polling_interval_ms | 10                                                                                                                                                                                               | hidden               | all 7 server(s) checked
 never_fsync                             | true                                                                                                                                                                                             | unsafe,advanced      | all 7 server(s) checked
 openssl_security_level_override         | 0                                                                                                                                                                                                | unsafe,hidden        | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports        | true                                                                                                                                                                                             | unsafe               | all 7 server(s) checked
 server_dump_info_format                 | pb                                                                                                                                                                                               | hidden               | all 7 server(s) checked
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-0/data/info.pb | hidden               | 127.22.208.129:40873
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-1/data/info.pb | hidden               | 127.22.208.130:36403
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-2/data/info.pb | hidden               | 127.22.208.131:39533
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-3/data/info.pb | hidden               | 127.22.208.132:35833
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-4/data/info.pb | hidden               | 127.22.208.133:35369
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-5/data/info.pb | hidden               | 127.22.208.134:44711
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.InsertAndDeleteWithCrashyNodes.1786521625833775-23362-0/raft_consensus-itest-crashy-nodes-cluster/ts-6/data/info.pb | hidden               | 127.22.208.135:33345

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:44797 | all 7 server(s) checked
 time_source         | builtin              | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
     Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
 test-workload | 7  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 1
 Tablets        | 1
 Replicas       | 7

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 08:00:43.816828 23362 log_verifier.cc:126] Checking tablet a12d50650402488683716909cc273d1e
I20260812 08:00:43.900445 23362 log_verifier.cc:177] Verified matching terms for 123 ops in tablet a12d50650402488683716909cc273d1e
I20260812 08:00:44.011376 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 28779
I20260812 08:00:44.029155 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 28950
I20260812 08:00:44.051270 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 29145
I20260812 08:00:44.063056 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 29375
I20260812 08:00:44.074046 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 29568
I20260812 08:00:44.084769 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 29738
I20260812 08:00:44.100775 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 29949
I20260812 08:00:44.111644 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 24024
2026-08-12T08:00:44Z chronyd exiting
[       OK ] RaftConsensusITest.InsertAndDeleteWithCrashyNodes (16523 ms)
[ RUN      ] RaftConsensusITest.TestAddRemoveServer
I20260812 08:00:44.147995 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:00:44.148072 23362 ts_itest-base.cc:116] --------------
I20260812 08:00:44.148084 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:00:44.148095 23362 ts_itest-base.cc:118] 3 replicas per TS
I20260812 08:00:44.148108 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:00:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:00:44Z Disabled control of system clock
I20260812 08:00:44.159446 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:37717
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:37717
--master_add_server_when_underreplicated=false
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260812 08:00:44.274004 30297 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:44.274348 30297 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:44.274425 30297 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:44.278759 30297 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:00:44.278898 30297 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:44.278944 30297 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:00:44.278991 30297 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:00:44.283592 30297 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:37717
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:44.285089 30297 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:44.286373 30297 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:44.292976 30309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.292974 30305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.292974 30306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:44.293260 30297 server_base.cc:1061] running on GCE node
I20260812 08:00:44.294080 30297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:44.295225 30297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:44.296468 30297 hybrid_clock.cc:648] HybridClock initialized: now 1786521644296441 us; error 40 us; skew 500 ppm
I20260812 08:00:44.298727 30297 webserver.cc:533] Webserver started at http://127.22.208.190:32889/ using document root <none> and password file <none>
I20260812 08:00:44.299363 30297 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:44.299458 30297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:44.299712 30297 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:44.301484 30297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7e0b62eb86c044ada41ffaef13f71887"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.302038 30297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7e0b62eb86c044ada41ffaef13f71887"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.305850 30297 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 08:00:44.308403 30315 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.309566 30297 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:00:44.309753 30297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7e0b62eb86c044ada41ffaef13f71887"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.309891 30297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:44.328308 30297 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:44.329123 30297 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:44.329357 30297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:44.337936 30297 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:37717
I20260812 08:00:44.337934 30373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:37717 every 8 connection(s)
I20260812 08:00:44.339187 30297 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:00:44.342327 30374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:44.347415 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 30297
I20260812 08:00:44.347553 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:00:44.347929 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887: Bootstrap starting.
I20260812 08:00:44.350890 30374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:44.351824 30374 log.cc:826] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:44.353672 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887: No bootstrap required, opened a new log
I20260812 08:00:44.356469 30374 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } }
I20260812 08:00:44.356673 30374 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:44.356736 30374 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0b62eb86c044ada41ffaef13f71887, State: Initialized, Role: FOLLOWER
I20260812 08:00:44.357161 30374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } }
I20260812 08:00:44.357280 30374 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:00:44.357370 30374 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:00:44.357483 30374 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:44.358599 30374 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } }
I20260812 08:00:44.358942 30374 leader_election.cc:304] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0b62eb86c044ada41ffaef13f71887; no voters: 
I20260812 08:00:44.359246 30374 leader_election.cc:290] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:00:44.359345 30380 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:44.359508 30380 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0b62eb86c044ada41ffaef13f71887, State: Running, Role: LEADER
I20260812 08:00:44.359824 30380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } }
I20260812 08:00:44.360473 30374 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:00:44.361603 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0b62eb86c044ada41ffaef13f71887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } } }
I20260812 08:00:44.362241 30382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0b62eb86c044ada41ffaef13f71887. Latest consensus state: current_term: 1 leader_uuid: "7e0b62eb86c044ada41ffaef13f71887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0b62eb86c044ada41ffaef13f71887" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 37717 } } }
I20260812 08:00:44.362340 30382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:44.361752 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887 [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:44.363096 30389 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:00:44.366447 30389 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:00:44.372854 30389 catalog_manager.cc:1383] Generated new cluster ID: f8629fd8950a463488b6bef1dbe3f134
I20260812 08:00:44.372987 30389 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:00:44.381148 30389 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:00:44.382422 30389 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:00:44.392220 30389 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 7e0b62eb86c044ada41ffaef13f71887: Generated new TSK 0
I20260812 08:00:44.392927 30389 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:00:44.397397 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--builtin_ntp_servers=127.22.208.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260812 08:00:44.516175 30401 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:44.516472 30401 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:44.516536 30401 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:44.516633 30401 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:44.520218 30401 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:44.520414 30401 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:00:44.525343 30401 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--never_fsync=true
--heap_profile_path=/tmp/kudu.30401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:44.526777 30401 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:44.528039 30401 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:44.535552 30407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.535547 30406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.535547 30409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:44.536262 30401 server_base.cc:1061] running on GCE node
I20260812 08:00:44.536738 30401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:44.537451 30401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:44.538688 30401 hybrid_clock.cc:648] HybridClock initialized: now 1786521644538657 us; error 45 us; skew 500 ppm
I20260812 08:00:44.540814 30401 webserver.cc:533] Webserver started at http://127.22.208.129:36229/ using document root <none> and password file <none>
I20260812 08:00:44.541467 30401 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:44.541560 30401 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:44.541787 30401 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:44.543514 30401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a344bb20bccc462cb08e0434af581dde"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.544026 30401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a344bb20bccc462cb08e0434af581dde"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.547760 30401 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20260812 08:00:44.550091 30416 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.551270 30401 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:00:44.551378 30401 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a344bb20bccc462cb08e0434af581dde"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.551515 30401 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:44.567950 30401 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:44.568751 30401 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:44.568994 30401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:44.569720 30401 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:44.571209 30401 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:44.571292 30401 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.571372 30401 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:44.571431 30401 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.582669 30401 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:36259
I20260812 08:00:44.582736 30547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:36259 every 8 connection(s)
I20260812 08:00:44.583907 30401 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:00:44.585955 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 30401
I20260812 08:00:44.586225 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:00:44.589561 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--builtin_ntp_servers=127.22.208.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260812 08:00:44.595589 30548 heartbeater.cc:344] Connected to a master server at 127.22.208.190:37717
I20260812 08:00:44.595917 30548 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:44.596676 30548 heartbeater.cc:507] Master 127.22.208.190:37717 requested a full tablet report, sending...
I20260812 08:00:44.599284 30333 ts_manager.cc:194] Registered new tserver with Master: a344bb20bccc462cb08e0434af581dde (127.22.208.129:36259)
I20260812 08:00:44.601509 30333 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:50845
W20260812 08:00:44.720130 30552 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:44.720393 30552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:44.720415 30552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:44.720489 30552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:44.724067 30552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:44.724200 30552 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:00:44.728448 30552 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--never_fsync=true
--heap_profile_path=/tmp/kudu.30552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:44.729797 30552 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:44.730953 30552 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:44.738040 30559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:44.738147 30552 server_base.cc:1061] running on GCE node
W20260812 08:00:44.738075 30557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.738081 30561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:44.738902 30552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:44.739617 30552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:44.740888 30552 hybrid_clock.cc:648] HybridClock initialized: now 1786521644740891 us; error 60 us; skew 500 ppm
I20260812 08:00:44.743259 30552 webserver.cc:533] Webserver started at http://127.22.208.130:38231/ using document root <none> and password file <none>
I20260812 08:00:44.743966 30552 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:44.744069 30552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:44.744326 30552 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:44.746120 30552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "def15a4d74a44771aad0f98795992922"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.746771 30552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "def15a4d74a44771aad0f98795992922"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.751099 30552 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:00:44.753865 30567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.755300 30552 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:44.755475 30552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "def15a4d74a44771aad0f98795992922"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.755602 30552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:44.768179 30552 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:44.768920 30552 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:44.769134 30552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:44.769750 30552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:44.770859 30552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:44.770938 30552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.771016 30552 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:44.771070 30552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.781693 30552 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:43477
I20260812 08:00:44.781788 30696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:43477 every 8 connection(s)
I20260812 08:00:44.782851 30552 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:00:44.789587 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 30552
I20260812 08:00:44.789808 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:00:44.792809 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--builtin_ntp_servers=127.22.208.148:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260812 08:00:44.793331 30697 heartbeater.cc:344] Connected to a master server at 127.22.208.190:37717
I20260812 08:00:44.793617 30697 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:44.794258 30697 heartbeater.cc:507] Master 127.22.208.190:37717 requested a full tablet report, sending...
I20260812 08:00:44.795656 30333 ts_manager.cc:194] Registered new tserver with Master: def15a4d74a44771aad0f98795992922 (127.22.208.130:43477)
I20260812 08:00:44.796490 30333 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:34805
W20260812 08:00:44.907262 30700 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:44.907575 30700 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:44.907626 30700 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:44.907724 30700 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:44.911286 30700 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:44.911486 30700 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:00:44.915570 30700 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:37717
--never_fsync=true
--heap_profile_path=/tmp/kudu.30700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:44.916810 30700 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:44.917948 30700 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:44.924757 30706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.924774 30707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:44.924913 30709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:44.925343 30700 server_base.cc:1061] running on GCE node
I20260812 08:00:44.925796 30700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:44.926472 30700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:44.927662 30700 hybrid_clock.cc:648] HybridClock initialized: now 1786521644927634 us; error 44 us; skew 500 ppm
I20260812 08:00:44.929811 30700 webserver.cc:533] Webserver started at http://127.22.208.131:33279/ using document root <none> and password file <none>
I20260812 08:00:44.930536 30700 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:44.930650 30700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:44.930912 30700 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:44.933022 30700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e8035f3995f7404f9f81b5287db2acb1"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.933634 30700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e8035f3995f7404f9f81b5287db2acb1"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.937470 30700 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20260812 08:00:44.940188 30715 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.941583 30700 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260812 08:00:44.941710 30700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e8035f3995f7404f9f81b5287db2acb1"
format_stamp: "Formatted at 2026-08-12 08:00:44 on dist-test-slave-dhph"
I20260812 08:00:44.941828 30700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:44.959638 30700 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:44.960431 30700 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:44.960680 30700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:44.961254 30700 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:44.962442 30700 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:44.962495 30700 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:44.962536 30700 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:44.962615 30700 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260812 08:00:44.973119 30700 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:34991
I20260812 08:00:44.973147 30846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:34991 every 8 connection(s)
I20260812 08:00:44.974287 30700 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:00:44.981412 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 30700
I20260812 08:00:44.981563 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAddRemoveServer.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:00:44.984169 30847 heartbeater.cc:344] Connected to a master server at 127.22.208.190:37717
I20260812 08:00:44.984450 30847 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:44.984978 30847 heartbeater.cc:507] Master 127.22.208.190:37717 requested a full tablet report, sending...
I20260812 08:00:44.986145 30332 ts_manager.cc:194] Registered new tserver with Master: e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991)
I20260812 08:00:44.986788 30332 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:51117
I20260812 08:00:44.996960 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:45.009354 30332 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37586:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 08:00:45.011997 30332 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:00:45.029906 30769 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccf5c49a13d47fba729340b64f2c233 (DEFAULT_TABLE table=TestTable [id=2234e9a344ac43eaa5e1799add858d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:45.030030 30472 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccf5c49a13d47fba729340b64f2c233 (DEFAULT_TABLE table=TestTable [id=2234e9a344ac43eaa5e1799add858d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:45.030151 30621 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccf5c49a13d47fba729340b64f2c233 (DEFAULT_TABLE table=TestTable [id=2234e9a344ac43eaa5e1799add858d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:45.031015 30769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccf5c49a13d47fba729340b64f2c233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:45.031015 30472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccf5c49a13d47fba729340b64f2c233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:45.031049 30621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccf5c49a13d47fba729340b64f2c233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:45.036558 30866 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Bootstrap starting.
I20260812 08:00:45.037837 30868 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Bootstrap starting.
I20260812 08:00:45.037837 30867 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Bootstrap starting.
I20260812 08:00:45.039569 30866 tablet_bootstrap.cc:654] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:45.040004 30867 tablet_bootstrap.cc:654] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:45.040004 30868 tablet_bootstrap.cc:654] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:45.040434 30866 log.cc:826] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:45.040802 30868 log.cc:826] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:45.040802 30867 log.cc:826] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:45.042399 30867 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: No bootstrap required, opened a new log
I20260812 08:00:45.042415 30868 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: No bootstrap required, opened a new log
I20260812 08:00:45.042415 30866 tablet_bootstrap.cc:492] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: No bootstrap required, opened a new log
I20260812 08:00:45.042656 30868 ts_tablet_manager.cc:1403] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20260812 08:00:45.042656 30866 ts_tablet_manager.cc:1403] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:00:45.042680 30867 ts_tablet_manager.cc:1403] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:00:45.045660 30867 raft_consensus.cc:359] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.045910 30867 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a344bb20bccc462cb08e0434af581dde, State: Initialized, Role: FOLLOWER
I20260812 08:00:45.045825 30868 raft_consensus.cc:359] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.045825 30866 raft_consensus.cc:359] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.046027 30868 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8035f3995f7404f9f81b5287db2acb1, State: Initialized, Role: FOLLOWER
I20260812 08:00:45.046036 30866 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def15a4d74a44771aad0f98795992922, State: Initialized, Role: FOLLOWER
I20260812 08:00:45.046314 30867 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.046360 30868 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.046360 30866 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.047174 30548 heartbeater.cc:499] Master 127.22.208.190:37717 was elected leader, sending a full tablet report...
I20260812 08:00:45.047415 30866 ts_tablet_manager.cc:1434] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:00:45.047501 30847 heartbeater.cc:499] Master 127.22.208.190:37717 was elected leader, sending a full tablet report...
I20260812 08:00:45.047776 30697 heartbeater.cc:499] Master 127.22.208.190:37717 was elected leader, sending a full tablet report...
I20260812 08:00:45.048405 30868 ts_tablet_manager.cc:1434] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20260812 08:00:45.048640 30867 ts_tablet_manager.cc:1434] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.003s
I20260812 08:00:45.064718 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:45.066641 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a344bb20bccc462cb08e0434af581dde to finish bootstrapping
I20260812 08:00:45.073567 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver def15a4d74a44771aad0f98795992922 to finish bootstrapping
I20260812 08:00:45.080026 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e8035f3995f7404f9f81b5287db2acb1 to finish bootstrapping
W20260812 08:00:45.085585 30549 tablet.cc:2406] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:45.090528 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:45.090874 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:45.091006 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:45.091993 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.093158 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 1 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991), a344bb20bccc462cb08e0434af581dde (127.22.208.129:36259)
I20260812 08:00:45.097698 30493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a344bb20bccc462cb08e0434af581dde"
I20260812 08:00:45.097810 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:45.098001 30493 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:45.098076 30793 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:45.099152 30493 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 1.
I20260812 08:00:45.099133 30793 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 1.
I20260812 08:00:45.099762 30570 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a344bb20bccc462cb08e0434af581dde, def15a4d74a44771aad0f98795992922, e8035f3995f7404f9f81b5287db2acb1; no voters: 
I20260812 08:00:45.100075 30875 raft_consensus.cc:2804] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:45.100379 30875 raft_consensus.cc:697] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 LEADER]: Becoming Leader. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:45.100781 30875 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.104286 30331 catalog_manager.cc:5719] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 reported cstate change: term changed from 0 to 1, leader changed from <none> to def15a4d74a44771aad0f98795992922 (127.22.208.130). New cstate: current_term: 1 leader_uuid: "def15a4d74a44771aad0f98795992922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } health_report { overall_health: UNKNOWN } } }
W20260812 08:00:45.225427 30849 tablet.cc:2406] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:00:45.284129 30698 tablet.cc:2406] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:45.536203 30875 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:45.543432 30884 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:45.557678 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 30297
I20260812 08:00:45.571802 23362 raft_consensus-itest.cc:484] Pausing 2 tablet servers in config of size 3
I20260812 08:00:45.695988 30645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:45.696233 30645 raft_consensus.cc:591] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 LEADER]: Received request to step down
I20260812 08:00:45.696331 30645 raft_consensus.cc:3055] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 LEADER]: Stepping down as leader of term 1
I20260812 08:00:45.696357 30645 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:45.696506 30645 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.696628 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:00:45.697623 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:45.697750 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:45.697801 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:00:45.698894 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
I20260812 08:00:45.699317 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 3 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991), a344bb20bccc462cb08e0434af581dde (127.22.208.129:36259)
I20260812 08:00:45.840332 23362 raft_consensus-itest.cc:515] Expected timeout encountered on election with weakened config: Timed out: Replica TabletServer: def15a4d74a44771aad0f98795992922, Rpc address: host: "127.22.208.130" port: 43477 is not leader after waiting for 0.100s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.22.208.130:43477 timed out after -0.038s (SENT)
I20260812 08:00:45.840448 23362 raft_consensus-itest.cc:518] Resuming 2 tablet servers in config of size 3
I20260812 08:00:45.842998 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:45.844054 30492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a344bb20bccc462cb08e0434af581dde"
I20260812 08:00:45.844146 30792 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:00:45.844205 30492 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 1 FOLLOWER]: Advancing to term 3
I20260812 08:00:45.845309 30792 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 3.
I20260812 08:00:45.845309 30492 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 3.
I20260812 08:00:45.845722 30571 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a344bb20bccc462cb08e0434af581dde, def15a4d74a44771aad0f98795992922; no voters: 
I20260812 08:00:45.845942 30879 raft_consensus.cc:2804] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:00:45.846007 30879 raft_consensus.cc:697] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Becoming Leader. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:45.846148 30879 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }
W20260812 08:00:45.848004 30697 heartbeater.cc:646] Failed to heartbeat to 127.22.208.190:37717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.208.190:37717: connect: Connection refused (error 111)
I20260812 08:00:45.866844 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:45.867090 30645 raft_consensus.cc:476] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Not starting forced leader election -- already a leader
I20260812 08:00:45.868499 23362 raft_consensus-itest.cc:530] Successful election with full config of size 3
I20260812 08:00:46.239318 30793 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 3 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260812 08:00:46.240186 30879 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260812 08:00:46.243222 30847 heartbeater.cc:646] Failed to heartbeat to 127.22.208.190:37717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.208.190:37717: connect: Connection refused (error 111)
I20260812 08:00:46.244087 30492 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 3 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260812 08:00:46.244607 30898 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46060 (request call id 43) took 650 ms (client timeout 100 ms). Trace:
I20260812 08:00:46.244675 30879 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260812 08:00:46.244943 30898 rpcz_store.cc:269] 0812 08:00:45.594706 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:45.594849 (+   143us) service_pool.cc:224] Handling call
0812 08:00:46.244577 (+649728us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:00:45.595383 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8ccf5c49a13d47fba729340b64f2c233
0812 08:00:45.595517 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:00:45.595529 (+    12us) write_op.cc:435] Acquired schema lock
0812 08:00:45.595531 (+     2us) tablet.cc:662] Decoding operations
0812 08:00:45.595661 (+   130us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:00:45.595672 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:00:45.595675 (+     3us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:00:45.595731 (+    56us) tablet.cc:701] Row locks acquired
0812 08:00:45.595732 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:00:45.595764 (+    32us) write_op.cc:270] Start()
0812 08:00:45.595790 (+    26us) write_op.cc:276] Timestamp: P: 1786521645595703 usec, L: 0
0812 08:00:45.595791 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:00:45.595936 (+   145us) log.cc:844] Serialized 174 byte log entry
0812 08:00:46.242864 (+646928us) op_driver.cc:464] REPLICATION: finished
0812 08:00:46.243232 (+   368us) write_op.cc:301] APPLY: starting
0812 08:00:46.243251 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:00:46.243286 (+    35us) tablet.cc:1370] finished BulkCheckPresence
0812 08:00:46.243303 (+    17us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:00:46.243395 (+    92us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:00:46.243434 (+    39us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 08:00:46.243443 (+     9us) write_op.cc:312] APPLY: finished
0812 08:00:46.243510 (+    67us) log.cc:844] Serialized 22 byte log entry
0812 08:00:46.243568 (+    58us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:00:46.244532 (+   964us) write_op.cc:454] Released schema lock
0812 08:00:46.244539 (+     7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":298,"mutex_wait_us":63,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":694,"prepare.run_wall_time_us":695,"raft.queue_time_us":244,"raft.run_cpu_time_us":768,"raft.run_wall_time_us":966,"replication_time_us":647030,"spinlock_wait_cycles":1156480,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:00:46.249476 30548 heartbeater.cc:646] Failed to heartbeat to 127.22.208.190:37717 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.208.190:37717: connect: Connection refused (error 111)
I20260812 08:00:46.249722 23362 raft_consensus-itest.cc:1482] Remove: Going from 3 to 2 replicas
I20260812 08:00:46.249786 23362 raft_consensus-itest.cc:1485] Removing tserver with uuid a344bb20bccc462cb08e0434af581dde
I20260812 08:00:46.250926 30645 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:46.252331 30793 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 3 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260812 08:00:46.253243 30904 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 08:00:46.255615 30884 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index -1 to 6, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:46.256516 30793 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index -1 to 6, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:46.260210 23362 raft_consensus-itest.cc:484] Pausing 1 tablet servers in config of size 2
I20260812 08:00:46.372992 30645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:46.373200 30645 raft_consensus.cc:591] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Received request to step down
I20260812 08:00:46.373330 30645 raft_consensus.cc:3055] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Stepping down as leader of term 3
I20260812 08:00:46.373380 30645 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:46.373566 30645 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:46.373677 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 3 FOLLOWER]: Advancing to term 4
I20260812 08:00:46.375186 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:46.375361 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:46.375497 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 4 FOLLOWER]: Advancing to term 5
I20260812 08:00:46.376477 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:46.376902 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 5 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991)
I20260812 08:00:46.516583 23362 raft_consensus-itest.cc:515] Expected timeout encountered on election with weakened config: Timed out: Replica TabletServer: def15a4d74a44771aad0f98795992922, Rpc address: host: "127.22.208.130" port: 43477 is not leader after waiting for 0.100s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.22.208.130:43477 timed out after -0.038s (SENT)
I20260812 08:00:46.516695 23362 raft_consensus-itest.cc:518] Resuming 1 tablet servers in config of size 2
I20260812 08:00:46.517941 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 5 candidate_status { last_received { term: 3 index: 7 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:46.518108 30792 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 3 FOLLOWER]: Advancing to term 5
I20260812 08:00:46.519176 30792 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 5.
I20260812 08:00:46.519492 30793 raft_consensus.cc:1240] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Rejecting Update request from peer def15a4d74a44771aad0f98795992922 for earlier term 3. Current term is 5. Ops: [3.7-3.7]
I20260812 08:00:46.519635 30570 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: def15a4d74a44771aad0f98795992922, e8035f3995f7404f9f81b5287db2acb1; no voters: 
I20260812 08:00:46.519959 30904 raft_consensus.cc:2804] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 FOLLOWER]: Leader election won for term 5
I20260812 08:00:46.519982 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 3.7 3.6
I20260812 08:00:46.520162 30904 raft_consensus.cc:697] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Becoming Leader. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:46.520344 30904 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:46.622920 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 5.8 3.6
I20260812 08:00:46.825748 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 5.8 3.6
I20260812 08:00:47.012732 30793 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260812 08:00:47.013341 30904 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260812 08:00:47.016307 30911 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46060 (request call id 61) took 744 ms (client timeout 100 ms). Trace:
W20260812 08:00:47.016413 30911 rpcz_store.cc:269] 0812 08:00:46.271931 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:46.272095 (+   164us) service_pool.cc:224] Handling call
0812 08:00:47.016291 (+744196us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:00:46.272759 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8ccf5c49a13d47fba729340b64f2c233
0812 08:00:46.272864 (+   105us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:00:46.272869 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:00:46.272871 (+     2us) tablet.cc:662] Decoding operations
0812 08:00:46.273132 (+   261us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:00:46.273142 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:00:46.273145 (+     3us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:00:46.273190 (+    45us) tablet.cc:701] Row locks acquired
0812 08:00:46.273191 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:00:46.273239 (+    48us) write_op.cc:270] Start()
0812 08:00:46.273268 (+    29us) write_op.cc:276] Timestamp: P: 1786521646273137 usec, L: 0
0812 08:00:46.273270 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:00:46.273436 (+   166us) log.cc:844] Serialized 174 byte log entry
0812 08:00:47.015779 (+742343us) op_driver.cc:464] REPLICATION: finished
0812 08:00:47.015966 (+   187us) write_op.cc:301] APPLY: starting
0812 08:00:47.015981 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:00:47.016010 (+    29us) tablet.cc:1370] finished BulkCheckPresence
0812 08:00:47.016033 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:00:47.016088 (+    55us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:00:47.016117 (+    29us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 08:00:47.016121 (+     4us) write_op.cc:312] APPLY: finished
0812 08:00:47.016188 (+    67us) log.cc:844] Serialized 22 byte log entry
0812 08:00:47.016229 (+    41us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:00:47.016250 (+    21us) write_op.cc:454] Released schema lock
0812 08:00:47.016256 (+     6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":131,"num_ops":1,"prepare.queue_time_us":155,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":824,"raft.queue_time_us":72,"raft.run_cpu_time_us":717,"raft.run_wall_time_us":717,"replication_time_us":742475,"spinlock_wait_cycles":2816,"thread_start_us":65,"threads_started":1}]]}
I20260812 08:00:47.129145 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:47.129462 30645 raft_consensus.cc:476] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Not starting forced leader election -- already a leader
I20260812 08:00:47.131297 23362 raft_consensus-itest.cc:530] Successful election with full config of size 2
I20260812 08:00:47.138037 23362 raft_consensus-itest.cc:1482] Remove: Going from 2 to 1 replicas
I20260812 08:00:47.138114 23362 raft_consensus-itest.cc:1485] Removing tserver with uuid e8035f3995f7404f9f81b5287db2acb1
I20260812 08:00:47.139014 30645 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 5.9, Last appended by leader: 7, Current term: 5, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:47.139784 30884 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Committing config change with OpId 5.10: config changed from index 6 to 10, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:47.144731 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:47.144897 30645 raft_consensus.cc:476] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Not starting forced leader election -- already a leader
I20260812 08:00:47.146420 23362 raft_consensus-itest.cc:530] Successful election with full config of size 1
I20260812 08:00:47.151051 23362 raft_consensus-itest.cc:1501] Add: Going from 1 to 2 replicas
I20260812 08:00:47.151137 23362 raft_consensus-itest.cc:1504] Adding tserver with uuid e8035f3995f7404f9f81b5287db2acb1
I20260812 08:00:47.152083 30645 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 5.11, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.153427 30793 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 5 index: 9. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20260812 08:00:47.153888 30884 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260812 08:00:47.155719 30793 raft_consensus.cc:872] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
W20260812 08:00:47.155853 30793 raft_consensus.cc:1579] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 NON_PARTICIPANT]: Could not prepare op 'term: 5 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.156184 30793 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 NON_PARTICIPANT]: Committing config change with OpId 5.10: config changed from index 6 to 10, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:47.158547 30793 raft_consensus.cc:1481] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260812 08:00:47.159907 30904 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Committing config change with OpId 5.12: config changed from index 10 to 12, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } }
I20260812 08:00:47.160495 30793 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Committing config change with OpId 5.12: config changed from index 10 to 12, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } }
I20260812 08:00:47.163425 23362 raft_consensus-itest.cc:484] Pausing 1 tablet servers in config of size 2
I20260812 08:00:47.276908 30645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:47.277117 30645 raft_consensus.cc:591] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Received request to step down
I20260812 08:00:47.277400 30645 raft_consensus.cc:3055] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Stepping down as leader of term 5
I20260812 08:00:47.277501 30645 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:47.277712 30645 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.13, Last appended by leader: 13, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.277843 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 5 FOLLOWER]: Advancing to term 6
I20260812 08:00:47.279084 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:47.279294 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:47.279438 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 6 FOLLOWER]: Advancing to term 7
I20260812 08:00:47.280408 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.281205 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 7 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991)
I20260812 08:00:47.421512 23362 raft_consensus-itest.cc:515] Expected timeout encountered on election with weakened config: Timed out: Replica TabletServer: def15a4d74a44771aad0f98795992922, Rpc address: host: "127.22.208.130" port: 43477 is not leader after waiting for 0.100s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.22.208.130:43477 timed out after -0.039s (SENT)
I20260812 08:00:47.421702 23362 raft_consensus-itest.cc:518] Resuming 1 tablet servers in config of size 2
I20260812 08:00:47.423024 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 7 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:47.423734 30792 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 5 FOLLOWER]: Advancing to term 7
I20260812 08:00:47.424880 30792 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 7.
I20260812 08:00:47.425097 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:47.425208 30570 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: def15a4d74a44771aad0f98795992922, e8035f3995f7404f9f81b5287db2acb1; no voters: 
I20260812 08:00:47.425273 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:47.425352 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:47.426152 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.426395 30915 raft_consensus.cc:2764] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260812 08:00:47.426576 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 8 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991)
I20260812 08:00:47.427146 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 8 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:47.427279 30792 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 7 FOLLOWER]: Advancing to term 8
I20260812 08:00:47.428274 30792 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 8.
I20260812 08:00:47.428634 30570 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: def15a4d74a44771aad0f98795992922, e8035f3995f7404f9f81b5287db2acb1; no voters: 
I20260812 08:00:47.428856 30915 raft_consensus.cc:2804] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 FOLLOWER]: Leader election won for term 8
I20260812 08:00:47.429013 30915 raft_consensus.cc:697] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 LEADER]: Becoming Leader. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:47.429173 30915 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.13, Last appended by leader: 13, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.430310 23362 raft_consensus-itest.cc:530] Successful election with full config of size 2
I20260812 08:00:47.816876 30792 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 8 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 5 index: 13. Preceding OpId from leader: term: 8 index: 14. (index mismatch)
I20260812 08:00:47.817620 30915 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 08:00:47.820628 30920 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46060 (request call id 89) took 645 ms (client timeout 100 ms). Trace:
W20260812 08:00:47.820784 30920 rpcz_store.cc:269] 0812 08:00:47.175527 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:47.175665 (+   138us) service_pool.cc:224] Handling call
0812 08:00:47.820610 (+644945us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:00:47.176190 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8ccf5c49a13d47fba729340b64f2c233
0812 08:00:47.176299 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:00:47.176303 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:00:47.176305 (+     2us) tablet.cc:662] Decoding operations
0812 08:00:47.176378 (+    73us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:00:47.176387 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:00:47.176389 (+     2us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:00:47.176426 (+    37us) tablet.cc:701] Row locks acquired
0812 08:00:47.176427 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:00:47.176466 (+    39us) write_op.cc:270] Start()
0812 08:00:47.176496 (+    30us) write_op.cc:276] Timestamp: P: 1786521647176396 usec, L: 0
0812 08:00:47.176498 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:00:47.176638 (+   140us) log.cc:844] Serialized 174 byte log entry
0812 08:00:47.819913 (+643275us) op_driver.cc:464] REPLICATION: finished
0812 08:00:47.820184 (+   271us) write_op.cc:301] APPLY: starting
0812 08:00:47.820208 (+    24us) tablet.cc:1367] starting BulkCheckPresence
0812 08:00:47.820247 (+    39us) tablet.cc:1370] finished BulkCheckPresence
0812 08:00:47.820266 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:00:47.820314 (+    48us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:00:47.820347 (+    33us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 08:00:47.820353 (+     6us) write_op.cc:312] APPLY: finished
0812 08:00:47.820419 (+    66us) log.cc:844] Serialized 22 byte log entry
0812 08:00:47.820518 (+    99us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:00:47.820565 (+    47us) write_op.cc:454] Released schema lock
0812 08:00:47.820571 (+     6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":205,"num_ops":1,"prepare.queue_time_us":167,"prepare.run_cpu_time_us":651,"prepare.run_wall_time_us":652,"raft.queue_time_us":99,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":358,"replication_time_us":643380,"spinlock_wait_cycles":98560,"thread_start_us":83,"threads_started":1}]]}
I20260812 08:00:47.826390 23362 raft_consensus-itest.cc:1501] Add: Going from 2 to 3 replicas
I20260812 08:00:47.826472 23362 raft_consensus-itest.cc:1504] Adding tserver with uuid a344bb20bccc462cb08e0434af581dde
I20260812 08:00:47.827795 30645 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 8.15, Last appended by leader: 13, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
I20260812 08:00:47.829097 30492 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 3 FOLLOWER]: Advancing to term 8
I20260812 08:00:47.829349 30792 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 8 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 8 index: 15. Preceding OpId from leader: term: 8 index: 16. (index mismatch)
I20260812 08:00:47.829843 30915 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260812 08:00:47.830201 30492 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 16. (index mismatch)
I20260812 08:00:47.830732 30915 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 08:00:47.832175 30922 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 LEADER]: Committing config change with OpId 8.16: config changed from index 12 to 16, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } } }
I20260812 08:00:47.832700 30792 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 8 FOLLOWER]: Committing config change with OpId 8.16: config changed from index 12 to 16, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } } }
I20260812 08:00:47.834908 30492 raft_consensus.cc:872] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
W20260812 08:00:47.835013 30492 raft_consensus.cc:1579] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Could not prepare op 'term: 5 index: 10' and following 6 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }
I20260812 08:00:47.835628 30492 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Committing config change with OpId 3.6: config changed from index -1 to 6, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:47.838816 30926 mvcc.cc:204] Tried to move back new op lower bound from 7317592666656116736 to 7317592664148480000. Current Snapshot: MvccSnapshot[applied={T|T < 7317592666656116736}]
I20260812 08:00:47.841887 23362 cluster_itest_util.cc:258] Not converged past 16 yet: 5.9 8.16 8.16
I20260812 08:00:47.842923 30492 raft_consensus.cc:1481] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260812 08:00:47.843451 30492 raft_consensus.cc:872] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
W20260812 08:00:47.843530 30492 raft_consensus.cc:1579] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Could not prepare op 'term: 5 index: 12' and following 4 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } } peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }: 
  New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }
I20260812 08:00:47.843843 30492 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Committing config change with OpId 5.10: config changed from index 6 to 10, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } }
I20260812 08:00:47.846793 30492 raft_consensus.cc:1481] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260812 08:00:47.847397 30492 raft_consensus.cc:872] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }: 
  New config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
W20260812 08:00:47.847484 30492 raft_consensus.cc:1579] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Could not prepare op 'term: 8 index: 16' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } }.
  Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } }: 
  New config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
I20260812 08:00:47.847836 30492 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Committing config change with OpId 5.12: config changed from index 10 to 12, VOTER e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } }
I20260812 08:00:47.851568 30492 raft_consensus.cc:1481] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260812 08:00:47.851966 30492 raft_consensus.cc:2955] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 FOLLOWER]: Committing config change with OpId 8.16: config changed from index 12 to 16, VOTER a344bb20bccc462cb08e0434af581dde (127.22.208.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } } }
I20260812 08:00:47.947062 23362 raft_consensus-itest.cc:484] Pausing 2 tablet servers in config of size 3
I20260812 08:00:48.070834 30645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:48.071064 30645 raft_consensus.cc:591] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 LEADER]: Received request to step down
I20260812 08:00:48.071169 30645 raft_consensus.cc:3055] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 LEADER]: Stepping down as leader of term 8
I20260812 08:00:48.071213 30645 raft_consensus.cc:740] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:48.071473 30645 consensus_queue.cc:260] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 8.17, Last appended by leader: 17, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
I20260812 08:00:48.071594 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 8 FOLLOWER]: Advancing to term 9
I20260812 08:00:48.072723 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:48.072871 30645 raft_consensus.cc:493] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:48.072974 30645 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 9 FOLLOWER]: Advancing to term 10
I20260812 08:00:48.073863 30645 raft_consensus.cc:515] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
I20260812 08:00:48.074337 30645 leader_election.cc:290] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 10 election: Requested vote from peers e8035f3995f7404f9f81b5287db2acb1 (127.22.208.131:34991), a344bb20bccc462cb08e0434af581dde (127.22.208.129:36259)
I20260812 08:00:48.213992 23362 raft_consensus-itest.cc:515] Expected timeout encountered on election with weakened config: Timed out: Replica TabletServer: def15a4d74a44771aad0f98795992922, Rpc address: host: "127.22.208.130" port: 43477 is not leader after waiting for 0.100s: Not found: Error connecting to replica: Timed out: GetConsensusState RPC to 127.22.208.130:43477 timed out after -0.039s (SENT)
I20260812 08:00:48.214090 23362 raft_consensus-itest.cc:518] Resuming 2 tablet servers in config of size 3
I20260812 08:00:48.215200 30493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 10 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: true dest_uuid: "a344bb20bccc462cb08e0434af581dde"
I20260812 08:00:48.215415 30493 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 8 FOLLOWER]: Advancing to term 10
I20260812 08:00:48.216190 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233" candidate_uuid: "def15a4d74a44771aad0f98795992922" candidate_term: 10 candidate_status { last_received { term: 8 index: 17 } } ignore_live_leader: true dest_uuid: "e8035f3995f7404f9f81b5287db2acb1"
I20260812 08:00:48.216322 30793 raft_consensus.cc:3060] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 8 FOLLOWER]: Advancing to term 10
I20260812 08:00:48.216543 30493 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 10.
I20260812 08:00:48.217042 30492 raft_consensus.cc:1240] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 10 FOLLOWER]: Rejecting Update request from peer def15a4d74a44771aad0f98795992922 for earlier term 8. Current term is 10. Ops: [8.17-8.17]
I20260812 08:00:48.217547 30793 raft_consensus.cc:2468] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def15a4d74a44771aad0f98795992922 in term 10.
I20260812 08:00:48.217857 30792 raft_consensus.cc:1240] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 10 FOLLOWER]: Rejecting Update request from peer def15a4d74a44771aad0f98795992922 for earlier term 8. Current term is 10. Ops: [8.17-8.17]
I20260812 08:00:48.217916 30571 leader_election.cc:304] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a344bb20bccc462cb08e0434af581dde, def15a4d74a44771aad0f98795992922; no voters: 
I20260812 08:00:48.218350 30915 raft_consensus.cc:2804] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 10 FOLLOWER]: Leader election won for term 10
I20260812 08:00:48.218447 30915 raft_consensus.cc:697] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 10 LEADER]: Becoming Leader. State: Replica: def15a4d74a44771aad0f98795992922, State: Running, Role: LEADER
I20260812 08:00:48.218609 30915 consensus_queue.cc:237] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 8.17, Last appended by leader: 17, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "def15a4d74a44771aad0f98795992922" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 43477 } } peers { permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { } } peers { permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { } }
I20260812 08:00:48.220450 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 8.16 8.16 10.18
I20260812 08:00:48.324081 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 8.16 8.16 10.18
I20260812 08:00:48.527961 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 8.16 8.16 10.18
I20260812 08:00:48.652791 30792 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P e8035f3995f7404f9f81b5287db2acb1 [term 10 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 8 index: 16. Preceding OpId from leader: term: 10 index: 18. (index mismatch)
I20260812 08:00:48.653463 30915 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8035f3995f7404f9f81b5287db2acb1" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 34991 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260812 08:00:48.657645 30492 raft_consensus.cc:1275] T 8ccf5c49a13d47fba729340b64f2c233 P a344bb20bccc462cb08e0434af581dde [term 10 FOLLOWER]: Refusing update from remote peer def15a4d74a44771aad0f98795992922: Log matching property violated. Preceding OpId in replica: term: 8 index: 16. Preceding OpId from leader: term: 10 index: 18. (index mismatch)
W20260812 08:00:48.658322 30935 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46060 (request call id 109) took 688 ms (client timeout 100 ms). Trace:
I20260812 08:00:48.658309 30904 consensus_queue.cc:1048] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a344bb20bccc462cb08e0434af581dde" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 36259 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
W20260812 08:00:48.658505 30935 rpcz_store.cc:269] 0812 08:00:47.969567 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:00:47.969668 (+   101us) service_pool.cc:224] Handling call
0812 08:00:48.658310 (+688642us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:00:47.970184 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8ccf5c49a13d47fba729340b64f2c233
0812 08:00:47.970296 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:00:47.970300 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:00:47.970302 (+     2us) tablet.cc:662] Decoding operations
0812 08:00:47.970382 (+    80us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:00:47.970391 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:00:47.970394 (+     3us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:00:47.970436 (+    42us) tablet.cc:701] Row locks acquired
0812 08:00:47.970437 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:00:47.970469 (+    32us) write_op.cc:270] Start()
0812 08:00:47.970496 (+    27us) write_op.cc:276] Timestamp: P: 1786521647970410 usec, L: 0
0812 08:00:47.970498 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:00:47.970736 (+   238us) log.cc:844] Serialized 174 byte log entry
0812 08:00:48.656394 (+685658us) op_driver.cc:464] REPLICATION: finished
0812 08:00:48.656950 (+   556us) write_op.cc:301] APPLY: starting
0812 08:00:48.656969 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:00:48.657003 (+    34us) tablet.cc:1370] finished BulkCheckPresence
0812 08:00:48.658048 (+  1045us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:00:48.658098 (+    50us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:00:48.658129 (+    31us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0812 08:00:48.658134 (+     5us) write_op.cc:312] APPLY: finished
0812 08:00:48.658196 (+    62us) log.cc:844] Serialized 22 byte log entry
0812 08:00:48.658255 (+    59us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:00:48.658268 (+    13us) write_op.cc:454] Released schema lock
0812 08:00:48.658273 (+     5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":299,"mutex_wait_us":1,"num_ops":1,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":713,"raft.queue_time_us":218,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":494,"replication_time_us":685829,"spinlock_wait_cycles":117632,"thread_start_us":86,"threads_started":1}]]}
I20260812 08:00:48.833472 30645 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccf5c49a13d47fba729340b64f2c233"
dest_uuid: "def15a4d74a44771aad0f98795992922"
 from {username='slave'} at 127.0.0.1:46060
I20260812 08:00:48.833750 30645 raft_consensus.cc:476] T 8ccf5c49a13d47fba729340b64f2c233 P def15a4d74a44771aad0f98795992922 [term 10 LEADER]: Not starting forced leader election -- already a leader
I20260812 08:00:48.835712 23362 raft_consensus-itest.cc:530] Successful election with full config of size 3
I20260812 08:00:48.842834 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 30401
I20260812 08:00:48.855309 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 30552
I20260812 08:00:48.865746 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 30700
2026-08-12T08:00:48Z chronyd exiting
[       OK ] RaftConsensusITest.TestAddRemoveServer (4746 ms)
[ RUN      ] RaftConsensusITest.TestAutoCreateReplica
I20260812 08:00:48.894523 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:00:48.894639 23362 ts_itest-base.cc:116] --------------
I20260812 08:00:48.894662 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:00:48.894686 23362 ts_itest-base.cc:118] 2 replicas per TS
I20260812 08:00:48.894703 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:00:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:00:48Z Disabled control of system clock
I20260812 08:00:48.906534 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:35311
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:35311
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false with env {}
W20260812 08:00:49.027989 30952 flags.cc:432] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20260812 08:00:49.028379 30952 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:49.028473 30952 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:49.028579 30952 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:49.033165 30952 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:00:49.033318 30952 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:49.033365 30952 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:00:49.033414 30952 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:00:49.038545 30952 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:35311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:49.040089 30952 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:49.041539 30952 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:49.049362 30959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.049398 30957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.049373 30962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:49.049938 30952 server_base.cc:1061] running on GCE node
I20260812 08:00:49.050856 30952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:49.052027 30952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:49.053344 30952 hybrid_clock.cc:648] HybridClock initialized: now 1786521649053326 us; error 42 us; skew 500 ppm
I20260812 08:00:49.055764 30952 webserver.cc:533] Webserver started at http://127.22.208.190:34227/ using document root <none> and password file <none>
I20260812 08:00:49.056428 30952 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:49.056530 30952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:49.056788 30952 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:49.058561 30952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "04d03a6fa3a84b9a9620117ce8f30c8e"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.059226 30952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "04d03a6fa3a84b9a9620117ce8f30c8e"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.063160 30952 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260812 08:00:49.065801 30968 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.067284 30952 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:49.067476 30952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "04d03a6fa3a84b9a9620117ce8f30c8e"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.067615 30952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:49.090924 30952 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:49.091743 30952 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:49.091957 30952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:49.100099 30952 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:35311
I20260812 08:00:49.100123 31030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:35311 every 8 connection(s)
I20260812 08:00:49.102771 30952 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:00:49.105115 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 30952
I20260812 08:00:49.105248 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:00:49.106356 31031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:49.112960 31031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e: Bootstrap starting.
I20260812 08:00:49.116019 31031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:49.117082 31031 log.cc:826] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:49.119498 31031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e: No bootstrap required, opened a new log
I20260812 08:00:49.123070 31031 raft_consensus.cc:359] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } }
I20260812 08:00:49.123348 31031 raft_consensus.cc:385] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:00:49.123445 31031 raft_consensus.cc:740] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d03a6fa3a84b9a9620117ce8f30c8e, State: Initialized, Role: FOLLOWER
I20260812 08:00:49.123940 31031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } }
I20260812 08:00:49.124102 31031 raft_consensus.cc:399] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:00:49.124195 31031 raft_consensus.cc:493] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:00:49.124342 31031 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:49.125268 31031 raft_consensus.cc:515] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } }
I20260812 08:00:49.125680 31031 leader_election.cc:304] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d03a6fa3a84b9a9620117ce8f30c8e; no voters: 
I20260812 08:00:49.126031 31031 leader_election.cc:290] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:00:49.126397 31037 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:49.127017 31037 raft_consensus.cc:697] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [term 1 LEADER]: Becoming Leader. State: Replica: 04d03a6fa3a84b9a9620117ce8f30c8e, State: Running, Role: LEADER
I20260812 08:00:49.127404 31037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } }
I20260812 08:00:49.127585 31031 sys_catalog.cc:565] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:00:49.129176 31039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d03a6fa3a84b9a9620117ce8f30c8e. Latest consensus state: current_term: 1 leader_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } } }
I20260812 08:00:49.129326 31038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d03a6fa3a84b9a9620117ce8f30c8e" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 35311 } } }
I20260812 08:00:49.129446 31039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:49.129451 31038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e [sys.catalog]: This master's current role is: LEADER
I20260812 08:00:49.129946 31049 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:00:49.133244 31049 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:00:49.139791 31049 catalog_manager.cc:1383] Generated new cluster ID: 14cb941804fc45da8d09951c97d36276
I20260812 08:00:49.139968 31049 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:00:49.151343 31049 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:00:49.153046 31049 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:00:49.164650 31049 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 04d03a6fa3a84b9a9620117ce8f30c8e: Generated new TSK 0
I20260812 08:00:49.165637 31049 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:00:49.176793 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35311
--builtin_ntp_servers=127.22.208.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--flush_threshold_mb=1
--maintenance_manager_polling_interval_ms=300 with env {}
W20260812 08:00:49.312084 31059 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:49.312422 31059 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:49.312494 31059 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:49.312604 31059 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:49.318150 31059 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:49.318493 31059 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260812 08:00:49.319334 31059 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:00:49.325938 31059 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--flush_threshold_mb=1
--tserver_master_addrs=127.22.208.190:35311
--never_fsync=true
--heap_profile_path=/tmp/kudu.31059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=300
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:49.327435 31059 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:49.328817 31059 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:00:49.338604 31067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.338825 31064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.338722 31065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:49.339023 31059 server_base.cc:1061] running on GCE node
I20260812 08:00:49.339484 31059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:49.340247 31059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:49.341490 31059 hybrid_clock.cc:648] HybridClock initialized: now 1786521649341458 us; error 52 us; skew 500 ppm
I20260812 08:00:49.344233 31059 webserver.cc:533] Webserver started at http://127.22.208.129:36299/ using document root <none> and password file <none>
I20260812 08:00:49.344879 31059 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:49.344974 31059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:49.345217 31059 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:49.348060 31059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.349041 31059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.353327 31059 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:00:49.356277 31073 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.357584 31059 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260812 08:00:49.357839 31059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.357986 31059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:49.373075 31059 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:49.373876 31059 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:49.374105 31059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:49.374841 31059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:49.375990 31059 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:49.376075 31059 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.376154 31059 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:49.376216 31059 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.388475 31059 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:44159
I20260812 08:00:49.388578 31198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:44159 every 8 connection(s)
I20260812 08:00:49.389679 31059 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:00:49.397351 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31059
I20260812 08:00:49.397497 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:00:49.400732 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35311
--builtin_ntp_servers=127.22.208.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--flush_threshold_mb=1
--maintenance_manager_polling_interval_ms=300 with env {}
I20260812 08:00:49.401876 31199 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35311
I20260812 08:00:49.402197 31199 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:49.403048 31199 heartbeater.cc:507] Master 127.22.208.190:35311 requested a full tablet report, sending...
I20260812 08:00:49.404966 30988 ts_manager.cc:194] Registered new tserver with Master: ea676c1bc52444a2abb6e5cd1dc47bb0 (127.22.208.129:44159)
I20260812 08:00:49.406512 30988 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:38759
W20260812 08:00:49.522841 31203 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:49.523106 31203 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:49.523129 31203 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:49.523258 31203 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:49.527081 31203 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:49.527199 31203 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260812 08:00:49.527251 31203 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:00:49.532232 31203 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--flush_threshold_mb=1
--tserver_master_addrs=127.22.208.190:35311
--never_fsync=true
--heap_profile_path=/tmp/kudu.31203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=300
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:49.533592 31203 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:49.534910 31203 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:49.543011 31203 server_base.cc:1061] running on GCE node
W20260812 08:00:49.542903 31211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.542904 31209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.542903 31208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:49.543803 31203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:49.544544 31203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:49.545816 31203 hybrid_clock.cc:648] HybridClock initialized: now 1786521649545800 us; error 37 us; skew 500 ppm
I20260812 08:00:49.548571 31203 webserver.cc:533] Webserver started at http://127.22.208.130:46747/ using document root <none> and password file <none>
I20260812 08:00:49.549336 31203 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:49.549439 31203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:49.549705 31203 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:49.551628 31203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e2ce98d1a0684ad4abec0703a4432229"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.552211 31203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e2ce98d1a0684ad4abec0703a4432229"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.556281 31203 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:00:49.558977 31217 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.560285 31203 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:00:49.560479 31203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e2ce98d1a0684ad4abec0703a4432229"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.560753 31203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:49.575053 31203 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:49.575776 31203 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:49.575986 31203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:49.576592 31203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:49.577647 31203 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:49.577730 31203 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.577803 31203 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:49.577854 31203 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.588672 31203 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:45169
I20260812 08:00:49.588685 31343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:45169 every 8 connection(s)
I20260812 08:00:49.589793 31203 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:00:49.599943 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31203
I20260812 08:00:49.600109 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:00:49.600608 31344 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35311
I20260812 08:00:49.600991 31344 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:49.601802 31344 heartbeater.cc:507] Master 127.22.208.190:35311 requested a full tablet report, sending...
I20260812 08:00:49.602581 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:35311
--builtin_ntp_servers=127.22.208.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--flush_threshold_mb=1
--maintenance_manager_polling_interval_ms=300 with env {}
I20260812 08:00:49.603384 30988 ts_manager.cc:194] Registered new tserver with Master: e2ce98d1a0684ad4abec0703a4432229 (127.22.208.130:45169)
I20260812 08:00:49.604151 30988 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:35637
W20260812 08:00:49.734727 31347 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 08:00:49.734987 31347 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:00:49.735009 31347 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:00:49.735057 31347 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:00:49.738538 31347 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:00:49.738654 31347 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260812 08:00:49.738708 31347 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:00:49.742811 31347 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--flush_threshold_mb=1
--tserver_master_addrs=127.22.208.190:35311
--never_fsync=true
--heap_profile_path=/tmp/kudu.31347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=300
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:00:49.744000 31347 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:00:49.745206 31347 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:00:49.752462 31347 server_base.cc:1061] running on GCE node
W20260812 08:00:49.752345 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.752346 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:00:49.752388 31356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:00:49.753055 31347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:00:49.753705 31347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:00:49.754923 31347 hybrid_clock.cc:648] HybridClock initialized: now 1786521649754897 us; error 42 us; skew 500 ppm
I20260812 08:00:49.757063 31347 webserver.cc:533] Webserver started at http://127.22.208.131:38583/ using document root <none> and password file <none>
I20260812 08:00:49.757694 31347 fs_manager.cc:362] Metadata directory not provided
I20260812 08:00:49.757786 31347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:00:49.758035 31347 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:00:49.760084 31347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a5ff4747fc77492ebdd5a71e1e50b362"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.760521 31347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a5ff4747fc77492ebdd5a71e1e50b362"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.765237 31347 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260812 08:00:49.767848 31362 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.769008 31347 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:00:49.769174 31347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a5ff4747fc77492ebdd5a71e1e50b362"
format_stamp: "Formatted at 2026-08-12 08:00:49 on dist-test-slave-dhph"
I20260812 08:00:49.769315 31347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:00:49.795658 31347 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:00:49.796411 31347 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:00:49.796622 31347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:00:49.797235 31347 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:00:49.798285 31347 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:00:49.798365 31347 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.798439 31347 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:00:49.798485 31347 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:00:49.808997 31347 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:41155
I20260812 08:00:49.809020 31487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:41155 every 8 connection(s)
I20260812 08:00:49.810406 31347 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:00:49.814550 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31347
I20260812 08:00:49.814755 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:00:49.821000 31488 heartbeater.cc:344] Connected to a master server at 127.22.208.190:35311
I20260812 08:00:49.821316 31488 heartbeater.cc:461] Registering TS with master...
I20260812 08:00:49.821919 31488 heartbeater.cc:507] Master 127.22.208.190:35311 requested a full tablet report, sending...
I20260812 08:00:49.823207 30988 ts_manager.cc:194] Registered new tserver with Master: a5ff4747fc77492ebdd5a71e1e50b362 (127.22.208.131:41155)
I20260812 08:00:49.823884 30988 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:51485
I20260812 08:00:49.830360 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:49.841662 30988 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46454:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20260812 08:00:49.862879 31125 tablet_service.cc:1511] Processing CreateTablet for tablet acad250398374c88bbbbf6eca6918652 (DEFAULT_TABLE table=TestTable [id=03660b5105ae4baa81bb6fbebfd9d420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:49.862879 31415 tablet_service.cc:1511] Processing CreateTablet for tablet acad250398374c88bbbbf6eca6918652 (DEFAULT_TABLE table=TestTable [id=03660b5105ae4baa81bb6fbebfd9d420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:00:49.864100 31415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acad250398374c88bbbbf6eca6918652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:49.864100 31125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acad250398374c88bbbbf6eca6918652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:00:49.871101 31504 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Bootstrap starting.
I20260812 08:00:49.872521 31505 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Bootstrap starting.
I20260812 08:00:49.874189 31504 tablet_bootstrap.cc:654] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:49.875002 31505 tablet_bootstrap.cc:654] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Neither blocks nor log segments found. Creating new log.
I20260812 08:00:49.875268 31504 log.cc:826] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:49.875886 31505 log.cc:826] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Log is configured to *not* fsync() on all Append() calls
I20260812 08:00:49.877573 31504 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: No bootstrap required, opened a new log
I20260812 08:00:49.877658 31505 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: No bootstrap required, opened a new log
I20260812 08:00:49.877825 31504 ts_tablet_manager.cc:1403] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20260812 08:00:49.877825 31505 ts_tablet_manager.cc:1403] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20260812 08:00:49.881520 31505 raft_consensus.cc:359] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.881520 31504 raft_consensus.cc:359] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.881815 31505 raft_consensus.cc:740] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ff4747fc77492ebdd5a71e1e50b362, State: Initialized, Role: FOLLOWER
I20260812 08:00:49.881827 31504 raft_consensus.cc:740] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea676c1bc52444a2abb6e5cd1dc47bb0, State: Initialized, Role: FOLLOWER
I20260812 08:00:49.882200 31504 consensus_queue.cc:260] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.882211 31505 consensus_queue.cc:260] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.883157 31488 heartbeater.cc:499] Master 127.22.208.190:35311 was elected leader, sending a full tablet report...
I20260812 08:00:49.883332 31504 ts_tablet_manager.cc:1434] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:00:49.883332 31505 ts_tablet_manager.cc:1434] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
I20260812 08:00:49.883754 31199 heartbeater.cc:499] Master 127.22.208.190:35311 was elected leader, sending a full tablet report...
I20260812 08:00:49.899950 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:00:49.901659 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ea676c1bc52444a2abb6e5cd1dc47bb0 to finish bootstrapping
I20260812 08:00:49.908198 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a5ff4747fc77492ebdd5a71e1e50b362 to finish bootstrapping
I20260812 08:00:49.919144 31150 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "acad250398374c88bbbbf6eca6918652"
dest_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0"
 from {username='slave'} at 127.0.0.1:60664
I20260812 08:00:49.919426 31150 raft_consensus.cc:493] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 08:00:49.919580 31150 raft_consensus.cc:3060] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:49.920572 31150 raft_consensus.cc:515] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.921635 31150 leader_election.cc:290] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [CANDIDATE]: Term 1 election: Requested vote from peers a5ff4747fc77492ebdd5a71e1e50b362 (127.22.208.131:41155)
I20260812 08:00:49.927392 31436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acad250398374c88bbbbf6eca6918652" candidate_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5ff4747fc77492ebdd5a71e1e50b362"
I20260812 08:00:49.927716 31436 raft_consensus.cc:3060] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:00:49.928931 31436 raft_consensus.cc:2468] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea676c1bc52444a2abb6e5cd1dc47bb0 in term 1.
I20260812 08:00:49.929601 31077 leader_election.cc:304] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a5ff4747fc77492ebdd5a71e1e50b362, ea676c1bc52444a2abb6e5cd1dc47bb0; no voters: 
I20260812 08:00:49.930161 31510 raft_consensus.cc:2804] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:00:49.930500 31510 raft_consensus.cc:697] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 1 LEADER]: Becoming Leader. State: Replica: ea676c1bc52444a2abb6e5cd1dc47bb0, State: Running, Role: LEADER
I20260812 08:00:49.930950 31510 consensus_queue.cc:237] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } }
I20260812 08:00:49.933951 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1
I20260812 08:00:49.935194 30987 catalog_manager.cc:5719] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea676c1bc52444a2abb6e5cd1dc47bb0 (127.22.208.129). New cstate: current_term: 1 leader_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } health_report { overall_health: HEALTHY } } }
W20260812 08:00:49.991199 31200 tablet.cc:2406] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:50.036631 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1
W20260812 08:00:50.111466 31489 tablet.cc:2406] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:00:50.239454 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 0.0 1.1
I20260812 08:00:50.372733 31510 consensus_queue.cc:1048] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:00:50.553982 23362 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20260812 08:00:50.554107 23362 raft_consensus-itest.cc:2015] Starting write workload...
I20260812 08:00:50.554951 23362 raft_consensus-itest.cc:2023] Only inserted 0 rows so far, sleeping for 100ms
I20260812 08:00:50.607023 31344 heartbeater.cc:499] Master 127.22.208.190:35311 was elected leader, sending a full tablet report...
I20260812 08:00:50.655174 23362 raft_consensus-itest.cc:2023] Only inserted 1000 rows so far, sleeping for 100ms
I20260812 08:00:50.755365 23362 raft_consensus-itest.cc:2023] Only inserted 2900 rows so far, sleeping for 100ms
I20260812 08:00:50.855700 23362 raft_consensus-itest.cc:2023] Only inserted 4500 rows so far, sleeping for 100ms
I20260812 08:00:50.955879 23362 raft_consensus-itest.cc:2023] Only inserted 6400 rows so far, sleeping for 100ms
I20260812 08:00:51.056064 23362 raft_consensus-itest.cc:2023] Only inserted 7500 rows so far, sleeping for 100ms
I20260812 08:00:51.156241 23362 raft_consensus-itest.cc:2023] Only inserted 8600 rows so far, sleeping for 100ms
I20260812 08:00:51.195694 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:51.256609 23362 raft_consensus-itest.cc:2023] Only inserted 10200 rows so far, sleeping for 100ms
I20260812 08:00:51.313637 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:51.356788 23362 raft_consensus-itest.cc:2023] Only inserted 11600 rows so far, sleeping for 100ms
I20260812 08:00:51.456959 23362 raft_consensus-itest.cc:2023] Only inserted 13100 rows so far, sleeping for 100ms
I20260812 08:00:51.558638 23362 raft_consensus-itest.cc:2023] Only inserted 14200 rows so far, sleeping for 100ms
I20260812 08:00:51.658816 23362 raft_consensus-itest.cc:2023] Only inserted 14900 rows so far, sleeping for 100ms
I20260812 08:00:51.710461 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.514s	user 0.261s	sys 0.008s Metrics: {"bytes_written":273795,"cfile_init":1,"compiler_manager_pool.queue_time_us":30577,"compiler_manager_pool.run_cpu_time_us":303772,"compiler_manager_pool.run_wall_time_us":477953,"dirs.queue_time_us":14993,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1645,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500,"spinlock_wait_cycles":273024,"thread_start_us":246,"threads_started":2}
I20260812 08:00:51.711934 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 85936 bytes on disk
I20260812 08:00:51.712719 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20260812 08:00:51.759002 23362 raft_consensus-itest.cc:2023] Only inserted 15900 rows so far, sleeping for 100ms
I20260812 08:00:51.834852 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.521s	user 0.243s	sys 0.004s Metrics: {"bytes_written":306926,"cfile_init":1,"dirs.queue_time_us":9708,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":3650,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1876,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10600,"thread_start_us":105,"threads_started":1}
I20260812 08:00:51.836083 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 96052 bytes on disk
I20260812 08:00:51.836894 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20260812 08:00:51.853554 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.560s	user 0.000s	sys 0.000s
I20260812 08:00:51.859190 23362 raft_consensus-itest.cc:2023] Only inserted 16700 rows so far, sleeping for 100ms
I20260812 08:00:51.959378 23362 raft_consensus-itest.cc:2023] Only inserted 18800 rows so far, sleeping for 100ms
I20260812 08:00:52.013618 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:52.059564 23362 raft_consensus-itest.cc:2023] Only inserted 20500 rows so far, sleeping for 100ms
I20260812 08:00:52.137626 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:52.159765 23362 raft_consensus-itest.cc:2023] Only inserted 21900 rows so far, sleeping for 100ms
I20260812 08:00:52.260685 23362 raft_consensus-itest.cc:2023] Only inserted 23200 rows so far, sleeping for 100ms
I20260812 08:00:52.360868 23362 raft_consensus-itest.cc:2023] Only inserted 24200 rows so far, sleeping for 100ms
I20260812 08:00:52.423514 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.410s	user 0.224s	sys 0.008s Metrics: {"bytes_written":306983,"cfile_init":1,"dirs.queue_time_us":47802,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1077,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1940,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10600,"thread_start_us":95,"threads_started":1}
I20260812 08:00:52.424873 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014695
I20260812 08:00:52.461061 23362 raft_consensus-itest.cc:2023] Only inserted 24900 rows so far, sleeping for 100ms
W20260812 08:00:52.559412 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.544s	user 0.001s	sys 0.000s
I20260812 08:00:52.561350 23362 raft_consensus-itest.cc:2023] Only inserted 25800 rows so far, sleeping for 100ms
I20260812 08:00:52.661525 23362 raft_consensus-itest.cc:2023] Only inserted 27200 rows so far, sleeping for 100ms
I20260812 08:00:52.761713 23362 raft_consensus-itest.cc:2023] Only inserted 28800 rows so far, sleeping for 100ms
I20260812 08:00:52.770694 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.633s	user 0.234s	sys 0.000s Metrics: {"bytes_written":325491,"cfile_init":1,"dirs.queue_time_us":6657,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":14005,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1897,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11300,"thread_start_us":94,"threads_started":1}
I20260812 08:00:52.771942 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 103604 bytes on disk
I20260812 08:00:52.772533 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20260812 08:00:52.773196 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014674
I20260812 08:00:52.861901 23362 raft_consensus-itest.cc:2023] Only inserted 30100 rows so far, sleeping for 100ms
I20260812 08:00:52.962097 23362 raft_consensus-itest.cc:2023] Only inserted 31200 rows so far, sleeping for 100ms
I20260812 08:00:53.062301 23362 raft_consensus-itest.cc:2023] Only inserted 32300 rows so far, sleeping for 100ms
I20260812 08:00:53.162499 23362 raft_consensus-itest.cc:2023] Only inserted 33300 rows so far, sleeping for 100ms
W20260812 08:00:53.224234 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.615s	user 0.000s	sys 0.000s
I20260812 08:00:53.262809 23362 raft_consensus-itest.cc:2023] Only inserted 34400 rows so far, sleeping for 100ms
W20260812 08:00:53.293445 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.532s	user 0.001s	sys 0.000s
I20260812 08:00:53.362978 23362 raft_consensus-itest.cc:2023] Only inserted 35000 rows so far, sleeping for 100ms
I20260812 08:00:53.463404 23362 raft_consensus-itest.cc:2023] Only inserted 36000 rows so far, sleeping for 100ms
I20260812 08:00:53.563581 23362 raft_consensus-itest.cc:2023] Only inserted 37300 rows so far, sleeping for 100ms
I20260812 08:00:53.593072 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.168s	user 0.385s	sys 0.000s Metrics: {"bytes_written":572343,"cfile_cache_hit":13,"cfile_cache_hit_bytes":471641,"cfile_cache_miss":39,"cfile_cache_miss_bytes":945823,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":7279,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1178,"drs_written":1,"lbm_read_time_us":1766,"lbm_reads_lt_1ms":63,"lbm_write_time_us":2765,"lbm_writes_lt_1ms":44,"num_input_rowsets":2,"peak_mem_usage":307809,"rows_written":20100,"thread_start_us":90,"threads_started":1}
I20260812 08:00:53.594465 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:53.663760 23362 raft_consensus-itest.cc:2023] Only inserted 38600 rows so far, sleeping for 100ms
I20260812 08:00:53.763937 23362 raft_consensus-itest.cc:2023] Only inserted 39600 rows so far, sleeping for 100ms
I20260812 08:00:53.864115 23362 raft_consensus-itest.cc:2023] Only inserted 40700 rows so far, sleeping for 100ms
I20260812 08:00:53.964296 23362 raft_consensus-itest.cc:2023] Only inserted 41600 rows so far, sleeping for 100ms
I20260812 08:00:53.984484 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.211s	user 0.406s	sys 0.004s Metrics: {"bytes_written":623350,"cfile_cache_hit":23,"cfile_cache_hit_bytes":852240,"cfile_cache_miss":31,"cfile_cache_miss_bytes":695006,"cfile_init":3,"delta_iterators_relevant":4,"dirs.queue_time_us":9496,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":6696,"drs_written":1,"lbm_read_time_us":1296,"lbm_reads_lt_1ms":43,"lbm_write_time_us":16833,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":45,"num_input_rowsets":2,"peak_mem_usage":335252,"rows_written":21900,"thread_start_us":85,"threads_started":1}
I20260812 08:00:53.996651 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.937278
I20260812 08:00:54.064476 23362 raft_consensus-itest.cc:2023] Only inserted 42500 rows so far, sleeping for 100ms
I20260812 08:00:54.164649 23362 raft_consensus-itest.cc:2023] Only inserted 43500 rows so far, sleeping for 100ms
I20260812 08:00:54.264839 23362 raft_consensus-itest.cc:2023] Only inserted 44500 rows so far, sleeping for 100ms
W20260812 08:00:54.308562 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.653s	user 0.001s	sys 0.000s
I20260812 08:00:54.366164 23362 raft_consensus-itest.cc:2023] Only inserted 45300 rows so far, sleeping for 100ms
I20260812 08:00:54.418790 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.824s	user 0.366s	sys 0.000s Metrics: {"bytes_written":505935,"cfile_init":1,"dirs.queue_time_us":11589,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2017,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2582,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17700,"thread_start_us":93,"threads_started":1}
I20260812 08:00:54.419683 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1046766 bytes of WAL
I20260812 08:00:54.420019 31078 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:00:54.420122 31078 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000001 (ops 1-326)
I20260812 08:00:54.420804 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:00:54.421144 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 352968 bytes on disk
I20260812 08:00:54.422009 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":197,"lbm_reads_lt_1ms":8}
I20260812 08:00:54.422606 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014583
I20260812 08:00:54.466341 23362 raft_consensus-itest.cc:2023] Only inserted 46200 rows so far, sleeping for 100ms
W20260812 08:00:54.534273 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.644s	user 0.001s	sys 0.000s
I20260812 08:00:54.566515 23362 raft_consensus-itest.cc:2023] Only inserted 47000 rows so far, sleeping for 100ms
I20260812 08:00:54.670287 23362 raft_consensus-itest.cc:2023] Only inserted 47600 rows so far, sleeping for 100ms
I20260812 08:00:54.770494 23362 raft_consensus-itest.cc:2023] Only inserted 47900 rows so far, sleeping for 100ms
I20260812 08:00:54.870708 23362 raft_consensus-itest.cc:2023] Only inserted 48700 rows so far, sleeping for 100ms
I20260812 08:00:54.970893 23362 raft_consensus-itest.cc:2023] Only inserted 49400 rows so far, sleeping for 100ms
I20260812 08:00:55.008642 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.012s	user 0.385s	sys 0.008s Metrics: {"bytes_written":567271,"cfile_init":1,"dirs.queue_time_us":67783,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2431,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15955,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19900,"thread_start_us":90,"threads_started":1}
I20260812 08:00:55.015831 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014542
I20260812 08:00:55.071063 23362 raft_consensus-itest.cc:2023] Only inserted 50200 rows so far, sleeping for 100ms
I20260812 08:00:55.096751 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.674s	user 0.641s	sys 0.008s Metrics: {"bytes_written":1074112,"cfile_cache_hit":12,"cfile_cache_hit_bytes":638477,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2032949,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":35696,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":3118,"drs_written":1,"lbm_read_time_us":3317,"lbm_reads_lt_1ms":82,"lbm_write_time_us":6079,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":577691,"rows_written":37800,"thread_start_us":566,"threads_started":1}
I20260812 08:00:55.103039 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:00:55.171241 23362 raft_consensus-itest.cc:2023] Only inserted 51400 rows so far, sleeping for 100ms
I20260812 08:00:55.271420 23362 raft_consensus-itest.cc:2023] Only inserted 53000 rows so far, sleeping for 100ms
I20260812 08:00:55.371608 23362 raft_consensus-itest.cc:2023] Only inserted 54400 rows so far, sleeping for 100ms
I20260812 08:00:55.471841 23362 raft_consensus-itest.cc:2023] Only inserted 55600 rows so far, sleeping for 100ms
I20260812 08:00:55.572016 23362 raft_consensus-itest.cc:2023] Only inserted 57300 rows so far, sleeping for 100ms
I20260812 08:00:55.672209 23362 raft_consensus-itest.cc:2023] Only inserted 58300 rows so far, sleeping for 100ms
I20260812 08:00:55.772382 23362 raft_consensus-itest.cc:2023] Only inserted 59200 rows so far, sleeping for 100ms
I20260812 08:00:55.793442 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.690s	user 0.269s	sys 0.000s Metrics: {"bytes_written":374419,"cfile_init":1,"dirs.queue_time_us":11134,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":2662,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":31732,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":13000,"thread_start_us":120,"threads_started":1}
I20260812 08:00:55.795807 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 484765 bytes on disk
I20260812 08:00:55.796772 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":123,"lbm_reads_lt_1ms":8}
I20260812 08:00:55.797379 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014314
I20260812 08:00:55.872538 23362 raft_consensus-itest.cc:2023] Only inserted 60700 rows so far, sleeping for 100ms
I20260812 08:00:55.972712 23362 raft_consensus-itest.cc:2023] Only inserted 62300 rows so far, sleeping for 100ms
W20260812 08:00:56.029175 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.561s	user 0.000s	sys 0.000s
I20260812 08:00:56.072916 23362 raft_consensus-itest.cc:2023] Only inserted 63400 rows so far, sleeping for 100ms
I20260812 08:00:56.173105 23362 raft_consensus-itest.cc:2023] Only inserted 64600 rows so far, sleeping for 100ms
I20260812 08:00:56.273466 23362 raft_consensus-itest.cc:2023] Only inserted 65600 rows so far, sleeping for 100ms
I20260812 08:00:56.373660 23362 raft_consensus-itest.cc:2023] Only inserted 66400 rows so far, sleeping for 100ms
W20260812 08:00:56.395763 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.549s	user 0.001s	sys 0.000s
I20260812 08:00:56.473845 23362 raft_consensus-itest.cc:2023] Only inserted 67200 rows so far, sleeping for 100ms
I20260812 08:00:56.574043 23362 raft_consensus-itest.cc:2023] Only inserted 67900 rows so far, sleeping for 100ms
I20260812 08:00:56.674238 23362 raft_consensus-itest.cc:2023] Only inserted 68400 rows so far, sleeping for 100ms
I20260812 08:00:56.717229 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.701s	user 0.690s	sys 0.017s Metrics: {"bytes_written":1186000,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1039926,"cfile_cache_miss":60,"cfile_cache_miss_bytes":1918362,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":12145,"dirs.run_cpu_time_us":373,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":3432,"lbm_reads_lt_1ms":88,"lbm_write_time_us":7590,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":68,"num_input_rowsets":2,"peak_mem_usage":638682,"rows_written":41800,"thread_start_us":242,"threads_started":1}
I20260812 08:00:56.718009 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1039466 bytes of WAL
I20260812 08:00:56.718279 31367 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:00:56.718389 31367 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000001 (ops 1-334)
I20260812 08:00:56.718998 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:00:56.719658 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.937278
I20260812 08:00:56.774446 23362 raft_consensus-itest.cc:2023] Only inserted 69100 rows so far, sleeping for 100ms
I20260812 08:00:56.874621 23362 raft_consensus-itest.cc:2023] Only inserted 69900 rows so far, sleeping for 100ms
I20260812 08:00:56.974814 23362 raft_consensus-itest.cc:2023] Only inserted 70700 rows so far, sleeping for 100ms
I20260812 08:00:57.075011 23362 raft_consensus-itest.cc:2023] Only inserted 71400 rows so far, sleeping for 100ms
I20260812 08:00:57.175211 23362 raft_consensus-itest.cc:2023] Only inserted 72300 rows so far, sleeping for 100ms
I20260812 08:00:57.278679 23362 raft_consensus-itest.cc:2023] Only inserted 73300 rows so far, sleeping for 100ms
I20260812 08:00:57.378868 23362 raft_consensus-itest.cc:2023] Only inserted 74100 rows so far, sleeping for 100ms
I20260812 08:00:57.479043 23362 raft_consensus-itest.cc:2023] Only inserted 75500 rows so far, sleeping for 100ms
I20260812 08:00:57.579226 23362 raft_consensus-itest.cc:2023] Only inserted 77000 rows so far, sleeping for 100ms
I20260812 08:00:57.679425 23362 raft_consensus-itest.cc:2023] Only inserted 77900 rows so far, sleeping for 100ms
I20260812 08:00:57.707988 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.984s	user 0.506s	sys 0.008s Metrics: {"bytes_written":765690,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":781,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3803,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":26900}
I20260812 08:00:57.708662 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014277
I20260812 08:00:57.779629 23362 raft_consensus-itest.cc:2023] Only inserted 79300 rows so far, sleeping for 100ms
I20260812 08:00:57.879884 23362 raft_consensus-itest.cc:2023] Only inserted 80200 rows so far, sleeping for 100ms
I20260812 08:00:57.921219 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 2.056s	user 0.850s	sys 0.008s Metrics: {"bytes_written":1435590,"cfile_cache_hit":13,"cfile_cache_hit_bytes":766574,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2843070,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":8690,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":3252,"drs_written":1,"lbm_read_time_us":4013,"lbm_reads_lt_1ms":91,"lbm_write_time_us":8435,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":77,"num_input_rowsets":2,"peak_mem_usage":775916,"rows_written":50800,"spinlock_wait_cycles":768,"thread_start_us":504,"threads_started":6}
I20260812 08:00:57.921993 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=2.929465
I20260812 08:00:57.980074 23362 raft_consensus-itest.cc:2023] Only inserted 81100 rows so far, sleeping for 100ms
I20260812 08:00:58.080269 23362 raft_consensus-itest.cc:2023] Only inserted 81900 rows so far, sleeping for 100ms
W20260812 08:00:58.106571 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.631s	user 0.001s	sys 0.000s
W20260812 08:00:58.124663 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.590s	user 0.001s	sys 0.000s
I20260812 08:00:58.180466 23362 raft_consensus-itest.cc:2023] Only inserted 82400 rows so far, sleeping for 100ms
I20260812 08:00:58.282650 23362 raft_consensus-itest.cc:2023] Only inserted 83100 rows so far, sleeping for 100ms
I20260812 08:00:58.382871 23362 raft_consensus-itest.cc:2023] Only inserted 83600 rows so far, sleeping for 100ms
I20260812 08:00:58.483062 23362 raft_consensus-itest.cc:2023] Only inserted 84400 rows so far, sleeping for 100ms
I20260812 08:00:58.583238 23362 raft_consensus-itest.cc:2023] Only inserted 85000 rows so far, sleeping for 100ms
I20260812 08:00:58.683425 23362 raft_consensus-itest.cc:2023] Only inserted 85500 rows so far, sleeping for 100ms
I20260812 08:00:58.731273 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.809s	user 0.587s	sys 0.009s Metrics: {"bytes_written":847233,"cfile_init":1,"dirs.queue_time_us":27030,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":7031,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4441,"lbm_writes_lt_1ms":55,"peak_mem_usage":0,"rows_written":29800,"thread_start_us":112,"threads_started":1}
I20260812 08:00:58.732048 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 785096 bytes on disk
I20260812 08:00:58.732683 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":121,"lbm_reads_lt_1ms":8}
I20260812 08:00:58.733486 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014155
I20260812 08:00:58.783625 23362 raft_consensus-itest.cc:2023] Only inserted 86000 rows so far, sleeping for 100ms
I20260812 08:00:58.886646 23362 raft_consensus-itest.cc:2023] Only inserted 86500 rows so far, sleeping for 100ms
I20260812 08:00:58.986829 23362 raft_consensus-itest.cc:2023] Only inserted 87200 rows so far, sleeping for 100ms
I20260812 08:00:59.087023 23362 raft_consensus-itest.cc:2023] Only inserted 87700 rows so far, sleeping for 100ms
I20260812 08:00:59.187438 23362 raft_consensus-itest.cc:2023] Only inserted 88200 rows so far, sleeping for 100ms
I20260812 08:00:59.287645 23362 raft_consensus-itest.cc:2023] Only inserted 88700 rows so far, sleeping for 100ms
I20260812 08:00:59.387837 23362 raft_consensus-itest.cc:2023] Only inserted 89400 rows so far, sleeping for 100ms
I20260812 08:00:59.461278 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.724s	user 1.179s	sys 0.016s Metrics: {"bytes_written":1940721,"cfile_cache_hit":23,"cfile_cache_hit_bytes":1898547,"cfile_cache_miss":83,"cfile_cache_miss_bytes":2990843,"cfile_init":4,"delta_iterators_relevant":4,"dirs.queue_time_us":25755,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":962,"drs_written":1,"lbm_read_time_us":44371,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":98,"lbm_write_time_us":11572,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1048853,"rows_written":68700,"thread_start_us":102,"threads_started":1}
I20260812 08:00:59.462071 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.937278
I20260812 08:00:59.488014 23362 raft_consensus-itest.cc:2023] Only inserted 90500 rows so far, sleeping for 100ms
I20260812 08:00:59.588203 23362 raft_consensus-itest.cc:2023] Only inserted 91800 rows so far, sleeping for 100ms
I20260812 08:00:59.688402 23362 raft_consensus-itest.cc:2023] Only inserted 93000 rows so far, sleeping for 100ms
I20260812 08:00:59.788599 23362 raft_consensus-itest.cc:2023] Only inserted 94100 rows so far, sleeping for 100ms
I20260812 08:00:59.888785 23362 raft_consensus-itest.cc:2023] Only inserted 94700 rows so far, sleeping for 100ms
I20260812 08:00:59.988984 23362 raft_consensus-itest.cc:2023] Only inserted 95400 rows so far, sleeping for 100ms
I20260812 08:01:00.089298 23362 raft_consensus-itest.cc:2023] Only inserted 96600 rows so far, sleeping for 100ms
W20260812 08:01:00.116986 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.618s	user 0.000s	sys 0.000s
I20260812 08:01:00.189498 23362 raft_consensus-itest.cc:2023] Only inserted 97300 rows so far, sleeping for 100ms
I20260812 08:01:00.289707 23362 raft_consensus-itest.cc:2023] Only inserted 98200 rows so far, sleeping for 100ms
I20260812 08:01:00.389922 23362 raft_consensus-itest.cc:2023] Only inserted 98900 rows so far, sleeping for 100ms
I20260812 08:01:00.490254 23362 raft_consensus-itest.cc:2023] Only inserted 99600 rows so far, sleeping for 100ms
I20260812 08:01:00.590713 23362 raft_consensus-itest.cc:2023] Only inserted 100600 rows so far, sleeping for 100ms
I20260812 08:01:00.622749 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.160s	user 0.428s	sys 0.004s Metrics: {"bytes_written":599992,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":870,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":39269,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":21000}
I20260812 08:01:00.623898 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1046604 bytes of WAL
I20260812 08:01:00.628437 31367 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:01:00.628540 31367 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000002 (ops 335-665)
I20260812 08:01:00.629427 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {}
I20260812 08:01:00.633407 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.014031
I20260812 08:01:00.690891 23362 raft_consensus-itest.cc:2023] Only inserted 101700 rows so far, sleeping for 100ms
I20260812 08:01:00.794626 23362 raft_consensus-itest.cc:2023] Only inserted 103000 rows so far, sleeping for 100ms
I20260812 08:01:00.894882 23362 raft_consensus-itest.cc:2023] Only inserted 104200 rows so far, sleeping for 100ms
I20260812 08:01:00.995079 23362 raft_consensus-itest.cc:2023] Only inserted 105600 rows so far, sleeping for 100ms
I20260812 08:01:01.095510 23362 raft_consensus-itest.cc:2023] Only inserted 107000 rows so far, sleeping for 100ms
I20260812 08:01:01.195717 23362 raft_consensus-itest.cc:2023] Only inserted 108100 rows so far, sleeping for 100ms
I20260812 08:01:01.295909 23362 raft_consensus-itest.cc:2023] Only inserted 108900 rows so far, sleeping for 100ms
W20260812 08:01:01.303344 31538 scanner-internal.cc:458] Time spent opening tablet: real 0.709s	user 0.000s	sys 0.000s
I20260812 08:01:01.396091 23362 raft_consensus-itest.cc:2023] Only inserted 110100 rows so far, sleeping for 100ms
I20260812 08:01:01.496284 23362 raft_consensus-itest.cc:2023] Only inserted 111300 rows so far, sleeping for 100ms
I20260812 08:01:01.596477 23362 raft_consensus-itest.cc:2023] Only inserted 112700 rows so far, sleeping for 100ms
I20260812 08:01:01.696676 23362 raft_consensus-itest.cc:2023] Only inserted 113400 rows so far, sleeping for 100ms
I20260812 08:01:01.796875 23362 raft_consensus-itest.cc:2023] Only inserted 114200 rows so far, sleeping for 100ms
I20260812 08:01:01.897183 23362 raft_consensus-itest.cc:2023] Only inserted 114800 rows so far, sleeping for 100ms
I20260812 08:01:01.936858 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 3.203s	user 1.351s	sys 0.049s Metrics: {"bytes_written":2272901,"cfile_cache_hit":12,"cfile_cache_hit_bytes":811479,"cfile_cache_miss":106,"cfile_cache_miss_bytes":4927363,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1261,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":4530,"drs_written":1,"lbm_read_time_us":7632,"lbm_reads_lt_1ms":126,"lbm_write_time_us":41411,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":113,"num_input_rowsets":2,"peak_mem_usage":1230303,"rows_written":80600,"thread_start_us":94,"threads_started":1}
I20260812 08:01:01.937597 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1047143 bytes of WAL
I20260812 08:01:01.938908 31078 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:01:01.938978 31078 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000002 (ops 327-651)
I20260812 08:01:01.939796 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:01:01.940337 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 819882 bytes on disk
I20260812 08:01:01.942028 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20260812 08:01:01.942845 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=2.929465
I20260812 08:01:01.997581 23362 raft_consensus-itest.cc:2023] Only inserted 115700 rows so far, sleeping for 100ms
I20260812 08:01:02.097899 23362 raft_consensus-itest.cc:2023] Only inserted 116500 rows so far, sleeping for 100ms
I20260812 08:01:02.198078 23362 raft_consensus-itest.cc:2023] Only inserted 117400 rows so far, sleeping for 100ms
I20260812 08:01:02.298270 23362 raft_consensus-itest.cc:2023] Only inserted 118500 rows so far, sleeping for 100ms
I20260812 08:01:02.398452 23362 raft_consensus-itest.cc:2023] Only inserted 119700 rows so far, sleeping for 100ms
I20260812 08:01:02.498626 23362 raft_consensus-itest.cc:2023] Only inserted 120800 rows so far, sleeping for 100ms
I20260812 08:01:02.598805 23362 raft_consensus-itest.cc:2023] Only inserted 121400 rows so far, sleeping for 100ms
I20260812 08:01:02.698985 23362 raft_consensus-itest.cc:2023] Only inserted 122000 rows so far, sleeping for 100ms
I20260812 08:01:02.799166 23362 raft_consensus-itest.cc:2023] Only inserted 122700 rows so far, sleeping for 100ms
I20260812 08:01:02.899363 23362 raft_consensus-itest.cc:2023] Only inserted 123400 rows so far, sleeping for 100ms
I20260812 08:01:02.999567 23362 raft_consensus-itest.cc:2023] Only inserted 124300 rows so far, sleeping for 100ms
I20260812 08:01:03.099776 23362 raft_consensus-itest.cc:2023] Only inserted 125300 rows so far, sleeping for 100ms
I20260812 08:01:03.200021 23362 raft_consensus-itest.cc:2023] Only inserted 126600 rows so far, sleeping for 100ms
I20260812 08:01:03.300209 23362 raft_consensus-itest.cc:2023] Only inserted 128400 rows so far, sleeping for 100ms
I20260812 08:01:03.400395 23362 raft_consensus-itest.cc:2023] Only inserted 129500 rows so far, sleeping for 100ms
I20260812 08:01:03.500589 23362 raft_consensus-itest.cc:2023] Only inserted 130300 rows so far, sleeping for 100ms
I20260812 08:01:03.600787 23362 raft_consensus-itest.cc:2023] Only inserted 131200 rows so far, sleeping for 100ms
I20260812 08:01:03.700973 23362 raft_consensus-itest.cc:2023] Only inserted 132100 rows so far, sleeping for 100ms
I20260812 08:01:03.741426 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 3.108s	user 1.563s	sys 0.012s Metrics: {"bytes_written":2524615,"cfile_cache_hit":35,"cfile_cache_hit_bytes":3956302,"cfile_cache_miss":95,"cfile_cache_miss_bytes":2453932,"cfile_init":4,"delta_iterators_relevant":4,"dirs.queue_time_us":1288,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":9665,"drs_written":1,"lbm_read_time_us":8373,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":110,"lbm_write_time_us":12638,"lbm_writes_lt_1ms":125,"num_input_rowsets":2,"peak_mem_usage":1369936,"rows_written":89700,"thread_start_us":92,"threads_started":1}
I20260812 08:01:03.742165 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 920113 bytes on disk
I20260812 08:01:03.743646 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20260812 08:01:03.744179 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=3.921653
I20260812 08:01:03.801154 23362 raft_consensus-itest.cc:2023] Only inserted 133000 rows so far, sleeping for 100ms
W20260812 08:01:03.872146 31537 scanner-internal.cc:458] Time spent opening tablet: real 0.630s	user 0.000s	sys 0.000s
I20260812 08:01:03.901336 23362 raft_consensus-itest.cc:2023] Only inserted 133700 rows so far, sleeping for 100ms
I20260812 08:01:04.001601 23362 raft_consensus-itest.cc:2023] Only inserted 134400 rows so far, sleeping for 100ms
I20260812 08:01:04.101791 23362 raft_consensus-itest.cc:2023] Only inserted 135100 rows so far, sleeping for 100ms
I20260812 08:01:04.201980 23362 raft_consensus-itest.cc:2023] Only inserted 135800 rows so far, sleeping for 100ms
I20260812 08:01:04.302170 23362 raft_consensus-itest.cc:2023] Only inserted 136600 rows so far, sleeping for 100ms
I20260812 08:01:04.402377 23362 raft_consensus-itest.cc:2023] Only inserted 137200 rows so far, sleeping for 100ms
I20260812 08:01:04.502616 23362 raft_consensus-itest.cc:2023] Only inserted 138400 rows so far, sleeping for 100ms
I20260812 08:01:04.605506 23362 raft_consensus-itest.cc:2023] Only inserted 139300 rows so far, sleeping for 100ms
I20260812 08:01:04.674361 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 2.731s	user 0.693s	sys 0.009s Metrics: {"bytes_written":995652,"cfile_init":1,"dirs.queue_time_us":74150,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":61251,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":33801,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":59,"peak_mem_usage":0,"rows_written":35100,"spinlock_wait_cycles":272128,"thread_start_us":109,"threads_started":1}
I20260812 08:01:04.675310 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1044008 bytes of WAL
I20260812 08:01:04.675624 31078 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:01:04.675730 31078 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000003 (ops 652-975)
I20260812 08:01:04.676309 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:01:04.676748 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.937278
I20260812 08:01:04.706090 23362 raft_consensus-itest.cc:2023] Only inserted 140300 rows so far, sleeping for 100ms
I20260812 08:01:04.806627 23362 raft_consensus-itest.cc:2023] Only inserted 140900 rows so far, sleeping for 100ms
I20260812 08:01:04.906926 23362 raft_consensus-itest.cc:2023] Only inserted 141600 rows so far, sleeping for 100ms
I20260812 08:01:05.007120 23362 raft_consensus-itest.cc:2023] Only inserted 142400 rows so far, sleeping for 100ms
I20260812 08:01:05.059158 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.300s	user 0.831s	sys 0.004s Metrics: {"bytes_written":1217517,"cfile_init":1,"dirs.queue_time_us":20223,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":952,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6836,"lbm_writes_lt_1ms":70,"peak_mem_usage":0,"rows_written":42900,"thread_start_us":170,"threads_started":1}
I20260812 08:01:05.060149 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.013617
I20260812 08:01:05.107318 23362 raft_consensus-itest.cc:2023] Only inserted 143600 rows so far, sleeping for 100ms
I20260812 08:01:05.208181 23362 raft_consensus-itest.cc:2023] Only inserted 144400 rows so far, sleeping for 100ms
I20260812 08:01:05.308374 23362 raft_consensus-itest.cc:2023] Only inserted 145300 rows so far, sleeping for 100ms
I20260812 08:01:05.408560 23362 raft_consensus-itest.cc:2023] Only inserted 145700 rows so far, sleeping for 100ms
I20260812 08:01:05.508754 23362 raft_consensus-itest.cc:2023] Only inserted 146700 rows so far, sleeping for 100ms
I20260812 08:01:05.609592 23362 raft_consensus-itest.cc:2023] Only inserted 148400 rows so far, sleeping for 100ms
I20260812 08:01:05.709785 23362 raft_consensus-itest.cc:2023] Only inserted 149400 rows so far, sleeping for 100ms
I20260812 08:01:05.809983 23362 raft_consensus-itest.cc:2027] Adding tserver with uuid e2ce98d1a0684ad4abec0703a4432229 as VOTER...
I20260812 08:01:05.840659 31150 consensus_queue.cc:237] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1505, Committed index: 1505, Last appended: 1.1506, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } }
I20260812 08:01:05.843357 31436 raft_consensus.cc:1275] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 1 FOLLOWER]: Refusing update from remote peer ea676c1bc52444a2abb6e5cd1dc47bb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1506. Preceding OpId from leader: term: 1 index: 1507. (index mismatch)
I20260812 08:01:05.843989 31637 consensus_queue.cc:1048] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1507, Last known committed idx: 1505, Time since last communication: 0.000s
I20260812 08:01:05.846200 31637 raft_consensus.cc:2955] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 [term 1 LEADER]: Committing config change with OpId 1.1507: config changed from index -1 to 1507, VOTER e2ce98d1a0684ad4abec0703a4432229 (127.22.208.130) added. New config: { opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } } }
I20260812 08:01:05.856248 31436 raft_consensus.cc:2955] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362 [term 1 FOLLOWER]: Committing config change with OpId 1.1507: config changed from index -1 to 1507, VOTER e2ce98d1a0684ad4abec0703a4432229 (127.22.208.130) added. New config: { opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } } }
I20260812 08:01:05.861630 30984 catalog_manager.cc:5719] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 reported cstate change: config changed from index -1 to 1507, VOTER e2ce98d1a0684ad4abec0703a4432229 (127.22.208.130) added. New cstate: current_term: 1 leader_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" committed_config { opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } health_report { overall_health: UNKNOWN } } }
W20260812 08:01:05.873363 31077 consensus_peers.cc:597] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0 -> Peer e2ce98d1a0684ad4abec0703a4432229 (127.22.208.130:45169): Couldn't send request to peer e2ce98d1a0684ad4abec0703a4432229. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acad250398374c88bbbbf6eca6918652. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:06.039194 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 1.362s	user 0.497s	sys 0.004s Metrics: {"bytes_written":694082,"cfile_init":1,"dirs.queue_time_us":17818,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":7934,"drs_written":1,"lbm_read_time_us":286,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5168,"lbm_writes_lt_1ms":49,"peak_mem_usage":0,"rows_written":24300,"thread_start_us":95,"threads_started":1}
I20260812 08:01:06.040376 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=2.022515
I20260812 08:01:06.272756 31656 ts_tablet_manager.cc:933] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Initiating tablet copy from peer ea676c1bc52444a2abb6e5cd1dc47bb0 (127.22.208.129:44159)
I20260812 08:01:06.274143 31656 tablet_copy_client.cc:323] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: tablet copy: Beginning tablet copy session from remote peer at address 127.22.208.129:44159
I20260812 08:01:06.288203 31172 tablet_copy_service.cc:140] P ea676c1bc52444a2abb6e5cd1dc47bb0: Received BeginTabletCopySession request for tablet acad250398374c88bbbbf6eca6918652 from peer e2ce98d1a0684ad4abec0703a4432229 ({username='slave'} at 127.22.208.130:34859)
I20260812 08:01:06.288427 31172 tablet_copy_service.cc:161] P ea676c1bc52444a2abb6e5cd1dc47bb0: Beginning new tablet copy session on tablet acad250398374c88bbbbf6eca6918652 from peer e2ce98d1a0684ad4abec0703a4432229 at {username='slave'} at 127.22.208.130:34859: session id = e2ce98d1a0684ad4abec0703a4432229-acad250398374c88bbbbf6eca6918652
I20260812 08:01:06.291396 31172 tablet_copy_source_session.cc:215] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Tablet Copy: opened 15 blocks and 2 log segments
I20260812 08:01:06.297763 31656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acad250398374c88bbbbf6eca6918652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:06.306715 31656 tablet_copy_client.cc:806] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: tablet copy: Starting download of 15 data blocks...
I20260812 08:01:06.361073 31656 tablet_copy_client.cc:670] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: tablet copy: Starting download of 2 WAL segments...
I20260812 08:01:06.371397 31656 tablet_copy_client.cc:538] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 08:01:06.375281 31656 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Bootstrap starting.
I20260812 08:01:06.492569 31656 log.cc:826] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:07.089937 31656 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Bootstrap replayed 1/2 log segments. Stats: ops{read=324 overwritten=0 applied=324 ignored=324} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 0 replicates
I20260812 08:01:07.777158 31656 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Bootstrap replayed 2/2 log segments. Stats: ops{read=541 overwritten=0 applied=541 ignored=426} inserts{seen=11400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 0 replicates
I20260812 08:01:07.777818 31656 tablet_bootstrap.cc:492] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Bootstrap complete.
I20260812 08:01:07.778841 31656 ts_tablet_manager.cc:1403] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Time spent bootstrapping tablet: real 1.404s	user 0.925s	sys 0.030s
I20260812 08:01:07.781903 31656 raft_consensus.cc:359] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } }
I20260812 08:01:07.782292 31656 raft_consensus.cc:740] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ce98d1a0684ad4abec0703a4432229, State: Initialized, Role: FOLLOWER
I20260812 08:01:07.782774 31656 consensus_queue.cc:260] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1516, Last appended: 1.1516, Last appended by leader: 1516, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1507 OBSOLETE_local: false peers { permanent_uuid: "a5ff4747fc77492ebdd5a71e1e50b362" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 41155 } } peers { permanent_uuid: "ea676c1bc52444a2abb6e5cd1dc47bb0" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 44159 } } peers { permanent_uuid: "e2ce98d1a0684ad4abec0703a4432229" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45169 } attrs { } }
I20260812 08:01:07.783926 31656 ts_tablet_manager.cc:1434] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20260812 08:01:07.790172 31172 tablet_copy_service.cc:342] P ea676c1bc52444a2abb6e5cd1dc47bb0: Request end of tablet copy session e2ce98d1a0684ad4abec0703a4432229-acad250398374c88bbbbf6eca6918652 received from {username='slave'} at 127.22.208.130:34859
I20260812 08:01:07.790331 31172 tablet_copy_service.cc:434] P ea676c1bc52444a2abb6e5cd1dc47bb0: ending tablet copy session e2ce98d1a0684ad4abec0703a4432229-acad250398374c88bbbbf6eca6918652 on tablet acad250398374c88bbbbf6eca6918652 with peer e2ce98d1a0684ad4abec0703a4432229
I20260812 08:01:07.921480 31345 maintenance_manager.cc:419] P e2ce98d1a0684ad4abec0703a4432229: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 1387628 bytes on disk
I20260812 08:01:07.923069 31222 maintenance_manager.cc:643] P e2ce98d1a0684ad4abec0703a4432229: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":204,"lbm_reads_lt_1ms":12}
I20260812 08:01:07.924876 31345 maintenance_manager.cc:419] P e2ce98d1a0684ad4abec0703a4432229: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=2.022515
I20260812 08:01:07.938156 31293 raft_consensus.cc:1217] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1506->[1.1507-1.1516]   Dedup: 1.1516->[]
I20260812 08:01:08.215199 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 3.155s	user 2.339s	sys 0.038s Metrics: {"bytes_written":3720464,"cfile_cache_hit":29,"cfile_cache_hit_bytes":4231773,"cfile_cache_miss":151,"cfile_cache_miss_bytes":5275855,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":12595,"dirs.run_cpu_time_us":494,"dirs.run_wall_time_us":2014,"drs_written":1,"lbm_read_time_us":10981,"lbm_reads_lt_1ms":175,"lbm_write_time_us":19468,"lbm_writes_lt_1ms":176,"num_input_rowsets":2,"peak_mem_usage":2045090,"rows_written":132600,"thread_start_us":380,"threads_started":2,"wal-append.queue_time_us":421}
I20260812 08:01:08.215904 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.937278
I20260812 08:01:08.642143 23362 raft_consensus-itest.cc:2038] Waiting for replicas to agree...
I20260812 08:01:08.659363 23362 raft_consensus-itest.cc:2049] Number of rows inserted: 151400
I20260812 08:01:08.661556 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.445s	user 0.429s	sys 0.000s Metrics: {"bytes_written":534433,"cfile_init":1,"dirs.queue_time_us":24328,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1077,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3303,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18800,"thread_start_us":132,"threads_started":1}
I20260812 08:01:08.662422 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 2085971 bytes of WAL
I20260812 08:01:08.662765 31367 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 2 log segments from log reader
I20260812 08:01:08.662887 31367 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000003 (ops 666-994)
I20260812 08:01:08.663009 31367 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P a5ff4747fc77492ebdd5a71e1e50b362: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000004 (ops 995-1322)
I20260812 08:01:08.664119 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {}
I20260812 08:01:08.664872 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 1572997 bytes on disk
I20260812 08:01:08.666220 31367 maintenance_manager.cc:643] P a5ff4747fc77492ebdd5a71e1e50b362: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":543,"lbm_reads_lt_1ms":8}
I20260812 08:01:08.667354 31489 maintenance_manager.cc:419] P a5ff4747fc77492ebdd5a71e1e50b362: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.013380
I20260812 08:01:08.753274 31273 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:01:08.769620 31415 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:01:08.770008 31125 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 04d03a6fa3a84b9a9620117ce8f30c8e | 127.22.208.190:35311 | HEALTHY

Unusual flags for Master:
                         Flag                         |                                                                                     Value                                                                                      |      Tags       |         Master
------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 allow_unsafe_replication_factor                      | true                                                                                                                                                                           | runtime,unsafe  | all 1 server(s) checked
 catalog_manager_evict_excess_replicas                | false                                                                                                                                                                          | runtime,hidden  | all 1 server(s) checked
 catalog_manager_wait_for_new_tablets_to_elect_leader | false                                                                                                                                                                          | hidden          | all 1 server(s) checked
 ipki_ca_key_size                                     | 768                                                                                                                                                                            | experimental    | all 1 server(s) checked
 ipki_server_key_size                                 | 768                                                                                                                                                                            | experimental    | all 1 server(s) checked
 never_fsync                                          | true                                                                                                                                                                           | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override                      | 0                                                                                                                                                                              | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                                        | true                                                                                                                                                                           | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports                     | true                                                                                                                                                                           | unsafe          | all 1 server(s) checked
 server_dump_info_format                              | pb                                                                                                                                                                             | hidden          | all 1 server(s) checked
 server_dump_info_path                                | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                                     | 512                                                                                                                                                                            | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:45683 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 a5ff4747fc77492ebdd5a71e1e50b362 | 127.22.208.131:41155 | HEALTHY | <none>   |       0        |       0
 e2ce98d1a0684ad4abec0703a4432229 | 127.22.208.130:45169 | HEALTHY | <none>   |       0        |       0
 ea676c1bc52444a2abb6e5cd1dc47bb0 | 127.22.208.129:44159 | HEALTHY | <none>   |       1        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                  Flag                   |                                                                                   Value                                                                                    |         Tags         |      Tablet Server
-----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
 enable_leader_failure_detection         | false                                                                                                                                                                      | unsafe               | all 3 server(s) checked
 flush_threshold_mb                      | 1                                                                                                                                                                          | runtime,experimental | all 3 server(s) checked
 ipki_server_key_size                    | 768                                                                                                                                                                        | experimental         | all 3 server(s) checked
 local_ip_for_outbound_sockets           | 127.22.208.129                                                                                                                                                             | experimental         | 127.22.208.129:44159
 local_ip_for_outbound_sockets           | 127.22.208.130                                                                                                                                                             | experimental         | 127.22.208.130:45169
 local_ip_for_outbound_sockets           | 127.22.208.131                                                                                                                                                             | experimental         | 127.22.208.131:41155
 maintenance_manager_polling_interval_ms | 300                                                                                                                                                                        | hidden               | all 3 server(s) checked
 never_fsync                             | true                                                                                                                                                                       | unsafe,advanced      | all 3 server(s) checked
 openssl_security_level_override         | 0                                                                                                                                                                          | unsafe,hidden        | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports        | true                                                                                                                                                                       | unsafe               | all 3 server(s) checked
 server_dump_info_format                 | pb                                                                                                                                                                         | hidden               | all 3 server(s) checked
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden               | 127.22.208.129:44159
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden               | 127.22.208.130:45169
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden               | 127.22.208.131:41155

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:45683 | all 3 server(s) checked
 time_source         | builtin              | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
   Name    | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 08:01:08.837240 23362 log_verifier.cc:126] Checking tablet acad250398374c88bbbbf6eca6918652
I20260812 08:01:09.175437 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 3.135s	user 2.650s	sys 0.028s Metrics: {"bytes_written":3925807,"cfile_cache_hit":19,"cfile_cache_hit_bytes":1350504,"cfile_cache_miss":183,"cfile_cache_miss_bytes":8655652,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":9666,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":2837,"drs_written":1,"lbm_read_time_us":11471,"lbm_reads_lt_1ms":219,"lbm_write_time_us":22622,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":184,"num_input_rowsets":3,"peak_mem_usage":2149474,"rows_written":140000,"thread_start_us":949,"threads_started":9,"wal-append.queue_time_us":274}
I20260812 08:01:09.176294 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1044272 bytes of WAL
I20260812 08:01:09.176657 31078 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:01:09.176724 31078 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P ea676c1bc52444a2abb6e5cd1dc47bb0: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000004 (ops 976-1299)
I20260812 08:01:09.177294 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:01:09.177618 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 1480351 bytes on disk
I20260812 08:01:09.178175 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20260812 08:01:09.178686 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
I20260812 08:01:09.194296 23362 log_verifier.cc:177] Verified matching terms for 542 ops in tablet acad250398374c88bbbbf6eca6918652
W20260812 08:01:09.721784 23362 scanner-internal.cc:458] Time spent opening tablet: real 0.516s	user 0.001s	sys 0.000s
I20260812 08:01:09.744535 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: FlushMRSOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.566s	user 0.277s	sys 0.004s Metrics: {"bytes_written":327827,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1273,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2321,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11400}
I20260812 08:01:09.745359 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 104528 bytes on disk
I20260812 08:01:09.745859 31078 maintenance_manager.cc:643] P ea676c1bc52444a2abb6e5cd1dc47bb0: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20260812 08:01:09.746395 31200 maintenance_manager.cc:419] P ea676c1bc52444a2abb6e5cd1dc47bb0: Scheduling CompactRowSetsOp(acad250398374c88bbbbf6eca6918652): perf score=1.013361
I20260812 08:01:10.617575 31222 maintenance_manager.cc:643] P e2ce98d1a0684ad4abec0703a4432229: CompactRowSetsOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 2.692s	user 2.481s	sys 0.051s Metrics: {"bytes_written":3925807,"cfile_cache_hit":15,"cfile_cache_hit_bytes":788406,"cfile_cache_miss":187,"cfile_cache_miss_bytes":9217750,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":9582,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":9497,"drs_written":1,"lbm_read_time_us":13093,"lbm_reads_lt_1ms":215,"lbm_write_time_us":23547,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":184,"mutex_wait_us":24,"num_input_rowsets":3,"peak_mem_usage":2149474,"rows_written":140000,"spinlock_wait_cycles":14592,"thread_start_us":345,"threads_started":3,"wal-append.queue_time_us":285}
I20260812 08:01:10.618531 31345 maintenance_manager.cc:419] P e2ce98d1a0684ad4abec0703a4432229: Scheduling LogGCOp(acad250398374c88bbbbf6eca6918652): free 1045048 bytes of WAL
I20260812 08:01:10.618896 31222 log_reader.cc:385] T acad250398374c88bbbbf6eca6918652: removed 1 log segments from log reader
I20260812 08:01:10.619009 31222 log.cc:1079] T acad250398374c88bbbbf6eca6918652 P e2ce98d1a0684ad4abec0703a4432229: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestAutoCreateReplica.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/acad250398374c88bbbbf6eca6918652/wal-000000001 (ops 976-1299)
I20260812 08:01:10.619542 31222 maintenance_manager.cc:643] P e2ce98d1a0684ad4abec0703a4432229: LogGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {}
I20260812 08:01:10.620025 31345 maintenance_manager.cc:419] P e2ce98d1a0684ad4abec0703a4432229: Scheduling UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652): 1480351 bytes on disk
I20260812 08:01:10.620550 31222 maintenance_manager.cc:643] P e2ce98d1a0684ad4abec0703a4432229: UndoDeltaBlockGCOp(acad250398374c88bbbbf6eca6918652) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260812 08:01:10.621050 31345 maintenance_manager.cc:419] P e2ce98d1a0684ad4abec0703a4432229: Scheduling FlushMRSOp(acad250398374c88bbbbf6eca6918652): perf score=1.000000
W20260812 08:01:10.805910 23362 client.cc:2042] Couldn't close scanner 3e4939d2a603492d903a75bee2b1a6a7: Aborted: reactor is shutting down (error 108)
I20260812 08:01:10.808655 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31059
I20260812 08:01:10.830735 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31203
I20260812 08:01:10.844527 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31347
I20260812 08:01:10.862287 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 30952
2026-08-12T08:01:10Z chronyd exiting
[       OK ] RaftConsensusITest.TestAutoCreateReplica (22010 ms)
[ RUN      ] RaftConsensusITest.TestHammerOneRow
I20260812 08:01:10.905514 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:01:10.905611 23362 ts_itest-base.cc:116] --------------
I20260812 08:01:10.905629 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:01:10.905649 23362 ts_itest-base.cc:118] 3 replicas per TS
I20260812 08:01:10.905660 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:01:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:01:10Z Disabled control of system clock
I20260812 08:01:10.919023 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:36717
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:36717 with env {}
W20260812 08:01:11.039260 31744 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:11.039592 31744 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:11.039683 31744 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:11.043298 31744 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:11.043424 31744 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:11.043463 31744 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:11.043506 31744 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:11.047796 31744 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:36717
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:11.049002 31744 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:11.050173 31744 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:11.056454 31751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.056429 31753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.056429 31750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:11.056782 31744 server_base.cc:1061] running on GCE node
I20260812 08:01:11.057394 31744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:11.058728 31744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:11.059930 31744 hybrid_clock.cc:648] HybridClock initialized: now 1786521671059894 us; error 66 us; skew 500 ppm
I20260812 08:01:11.067687 31744 webserver.cc:533] Webserver started at http://127.22.208.190:33759/ using document root <none> and password file <none>
I20260812 08:01:11.068400 31744 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:11.068475 31744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:11.068714 31744 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:11.070497 31744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3031281b6b6f431fb18db49ad31f7fb9"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.071139 31744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3031281b6b6f431fb18db49ad31f7fb9"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.075058 31744 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260812 08:01:11.077759 31761 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.079169 31744 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:01:11.079375 31744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3031281b6b6f431fb18db49ad31f7fb9"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.079519 31744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:11.090829 31744 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:11.091610 31744 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:11.091872 31744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:11.100637 31744 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:36717
I20260812 08:01:11.100777 31822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:36717 every 8 connection(s)
I20260812 08:01:11.101974 31744 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:01:11.105158 31823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:11.107712 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31744
I20260812 08:01:11.107900 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:01:11.111337 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: Bootstrap starting.
I20260812 08:01:11.114640 31823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:11.115477 31823 log.cc:826] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:11.117292 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: No bootstrap required, opened a new log
I20260812 08:01:11.120173 31823 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } }
I20260812 08:01:11.120417 31823 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:11.120508 31823 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3031281b6b6f431fb18db49ad31f7fb9, State: Initialized, Role: FOLLOWER
I20260812 08:01:11.120980 31823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } }
I20260812 08:01:11.121137 31823 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:01:11.121228 31823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:01:11.121344 31823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:11.122224 31823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } }
I20260812 08:01:11.122845 31823 leader_election.cc:304] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3031281b6b6f431fb18db49ad31f7fb9; no voters: 
I20260812 08:01:11.123255 31823 leader_election.cc:290] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:01:11.123481 31829 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:11.123723 31829 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [term 1 LEADER]: Becoming Leader. State: Replica: 3031281b6b6f431fb18db49ad31f7fb9, State: Running, Role: LEADER
I20260812 08:01:11.124294 31829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } }
I20260812 08:01:11.124707 31823 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:11.126441 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3031281b6b6f431fb18db49ad31f7fb9. Latest consensus state: current_term: 1 leader_uuid: "3031281b6b6f431fb18db49ad31f7fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } } }
I20260812 08:01:11.126843 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:11.127410 31830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3031281b6b6f431fb18db49ad31f7fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3031281b6b6f431fb18db49ad31f7fb9" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 36717 } } }
I20260812 08:01:11.127791 31830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9 [sys.catalog]: This master's current role is: LEADER
W20260812 08:01:11.130203 31845 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:11.130302 31845 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:11.130401 31844 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:01:11.131283 31844 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:01:11.139503 31844 catalog_manager.cc:1383] Generated new cluster ID: 6b268d561b1747b9901078f28bec3520
I20260812 08:01:11.139618 31844 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:01:11.173414 31844 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:01:11.174351 31844 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:01:11.183252 31844 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 3031281b6b6f431fb18db49ad31f7fb9: Generated new TSK 0
I20260812 08:01:11.184093 31844 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:01:11.188957 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--builtin_ntp_servers=127.22.208.148:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260812 08:01:11.324769 31849 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:11.325127 31849 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:11.325206 31849 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:11.333928 31849 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:11.334168 31849 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:01:11.342041 31849 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--never_fsync=true
--heap_profile_path=/tmp/kudu.31849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:11.343693 31849 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:11.345567 31849 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:11.354928 31855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.355458 31854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.355739 31857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:11.355867 31849 server_base.cc:1061] running on GCE node
I20260812 08:01:11.356438 31849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:11.357097 31849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:11.358305 31849 hybrid_clock.cc:648] HybridClock initialized: now 1786521671358290 us; error 33 us; skew 500 ppm
I20260812 08:01:11.361085 31849 webserver.cc:533] Webserver started at http://127.22.208.129:41567/ using document root <none> and password file <none>
I20260812 08:01:11.361964 31849 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:11.362061 31849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:11.362447 31849 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:11.364940 31849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "37575efb04814de79c4ef62b1a472a69"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.365583 31849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "37575efb04814de79c4ef62b1a472a69"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.372535 31849 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.004s
I20260812 08:01:11.375890 31866 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.377194 31849 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:01:11.377349 31849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "37575efb04814de79c4ef62b1a472a69"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.377486 31849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:11.403069 31849 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:11.403806 31849 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:11.404012 31849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:11.404649 31849 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:11.405828 31849 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:11.405905 31849 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.405978 31849 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:11.406023 31849 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.418253 31849 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:37473
I20260812 08:01:11.418286 31993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:37473 every 8 connection(s)
I20260812 08:01:11.419384 31849 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:01:11.420738 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31849
I20260812 08:01:11.420887 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:01:11.423976 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--builtin_ntp_servers=127.22.208.148:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 08:01:11.431629 31994 heartbeater.cc:344] Connected to a master server at 127.22.208.190:36717
I20260812 08:01:11.431998 31994 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:11.433535 31994 heartbeater.cc:507] Master 127.22.208.190:36717 requested a full tablet report, sending...
I20260812 08:01:11.436017 31781 ts_manager.cc:194] Registered new tserver with Master: 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473)
I20260812 08:01:11.437368 31781 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:41133
W20260812 08:01:11.547793 31997 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:11.548060 31997 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:11.548107 31997 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:11.551759 31997 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:11.551892 31997 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:01:11.556389 31997 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--never_fsync=true
--heap_profile_path=/tmp/kudu.31997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:11.557602 31997 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:11.558887 31997 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:11.566675 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.566658 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:11.566788 31997 server_base.cc:1061] running on GCE node
W20260812 08:01:11.566658 32005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:11.567487 31997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:11.568181 31997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:11.569410 31997 hybrid_clock.cc:648] HybridClock initialized: now 1786521671569383 us; error 38 us; skew 500 ppm
I20260812 08:01:11.571782 31997 webserver.cc:533] Webserver started at http://127.22.208.130:35675/ using document root <none> and password file <none>
I20260812 08:01:11.572427 31997 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:11.572521 31997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:11.572755 31997 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:11.574512 31997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ddcf1211ef9942bf910d906bbbe02475"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.575096 31997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ddcf1211ef9942bf910d906bbbe02475"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.578874 31997 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:01:11.581490 32014 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.582851 31997 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:01:11.582973 31997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ddcf1211ef9942bf910d906bbbe02475"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.583115 31997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:11.594039 31997 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:11.594859 31997 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:11.595074 31997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:11.595774 31997 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:11.597119 31997 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:11.597177 31997 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.597220 31997 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:11.597237 31997 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.608994 31997 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:45345
I20260812 08:01:11.609016 32142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:45345 every 8 connection(s)
I20260812 08:01:11.610172 31997 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:01:11.614562 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 31997
I20260812 08:01:11.614740 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:01:11.617660 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--builtin_ntp_servers=127.22.208.148:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260812 08:01:11.621886 32143 heartbeater.cc:344] Connected to a master server at 127.22.208.190:36717
I20260812 08:01:11.622187 32143 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:11.622879 32143 heartbeater.cc:507] Master 127.22.208.190:36717 requested a full tablet report, sending...
I20260812 08:01:11.624362 31781 ts_manager.cc:194] Registered new tserver with Master: ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345)
I20260812 08:01:11.625257 31781 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:56417
W20260812 08:01:11.744976 32146 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:11.745250 32146 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:11.745296 32146 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:11.748807 32146 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:11.748945 32146 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:01:11.753083 32146 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:36717
--never_fsync=true
--heap_profile_path=/tmp/kudu.32146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:11.754469 32146 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:11.755734 32146 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:11.763454 32155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.763588 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:11.763717 32152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:11.763967 32146 server_base.cc:1061] running on GCE node
I20260812 08:01:11.764379 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:11.765009 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:11.766212 32146 hybrid_clock.cc:648] HybridClock initialized: now 1786521671766189 us; error 52 us; skew 500 ppm
I20260812 08:01:11.768527 32146 webserver.cc:533] Webserver started at http://127.22.208.131:39313/ using document root <none> and password file <none>
I20260812 08:01:11.769134 32146 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:11.769196 32146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:11.769443 32146 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:11.771256 32146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dba6326e5e2b4938880eac572a3f5290"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.771756 32146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dba6326e5e2b4938880eac572a3f5290"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.775566 32146 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:01:11.778295 32162 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.779603 32146 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:01:11.779763 32146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dba6326e5e2b4938880eac572a3f5290"
format_stamp: "Formatted at 2026-08-12 08:01:11 on dist-test-slave-dhph"
I20260812 08:01:11.779881 32146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:11.796707 32146 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:11.797535 32146 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:11.797791 32146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:11.798730 32146 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:11.800019 32146 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:11.800097 32146 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.800173 32146 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:11.800225 32146 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:11.812933 32146 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:36487
I20260812 08:01:11.813035 32285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:36487 every 8 connection(s)
I20260812 08:01:11.814224 32146 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:01:11.818388 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 32146
I20260812 08:01:11.818547 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:01:11.824962 32286 heartbeater.cc:344] Connected to a master server at 127.22.208.190:36717
I20260812 08:01:11.825284 32286 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:11.825928 32286 heartbeater.cc:507] Master 127.22.208.190:36717 requested a full tablet report, sending...
I20260812 08:01:11.827366 31781 ts_manager.cc:194] Registered new tserver with Master: dba6326e5e2b4938880eac572a3f5290 (127.22.208.131:36487)
I20260812 08:01:11.828368 31781 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:46561
I20260812 08:01:11.835088 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:01:11.847716 31781 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46328:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 08:01:11.850245 31781 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:01:11.868963 32213 tablet_service.cc:1511] Processing CreateTablet for tablet 049a7ef1624e4420952a5c0960ea2492 (DEFAULT_TABLE table=TestTable [id=7cf5cfba19f24ebca81cfca7875198b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:01:11.868994 31920 tablet_service.cc:1511] Processing CreateTablet for tablet 049a7ef1624e4420952a5c0960ea2492 (DEFAULT_TABLE table=TestTable [id=7cf5cfba19f24ebca81cfca7875198b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:01:11.869310 32068 tablet_service.cc:1511] Processing CreateTablet for tablet 049a7ef1624e4420952a5c0960ea2492 (DEFAULT_TABLE table=TestTable [id=7cf5cfba19f24ebca81cfca7875198b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:01:11.869961 32213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a7ef1624e4420952a5c0960ea2492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:11.870002 31920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a7ef1624e4420952a5c0960ea2492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:11.870118 32068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049a7ef1624e4420952a5c0960ea2492. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:11.875905 32306 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Bootstrap starting.
I20260812 08:01:11.876740 32308 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Bootstrap starting.
I20260812 08:01:11.878965 32308 tablet_bootstrap.cc:654] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:11.878961 32306 tablet_bootstrap.cc:654] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:11.879004 32307 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Bootstrap starting.
I20260812 08:01:11.879777 32308 log.cc:826] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:11.880016 32306 log.cc:826] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:11.881714 32306 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: No bootstrap required, opened a new log
I20260812 08:01:11.881893 32307 tablet_bootstrap.cc:654] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:11.881947 32308 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: No bootstrap required, opened a new log
I20260812 08:01:11.881973 32306 ts_tablet_manager.cc:1403] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 08:01:11.882131 32308 ts_tablet_manager.cc:1403] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20260812 08:01:11.882836 32307 log.cc:826] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:11.885039 32307 tablet_bootstrap.cc:492] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: No bootstrap required, opened a new log
I20260812 08:01:11.885277 32307 ts_tablet_manager.cc:1403] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.005s
I20260812 08:01:11.885540 32308 raft_consensus.cc:359] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.885813 32308 raft_consensus.cc:385] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:11.885903 32308 raft_consensus.cc:740] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcf1211ef9942bf910d906bbbe02475, State: Initialized, Role: FOLLOWER
I20260812 08:01:11.886415 32308 consensus_queue.cc:260] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.886756 32306 raft_consensus.cc:359] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.887053 32306 raft_consensus.cc:385] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:11.887208 32306 raft_consensus.cc:740] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba6326e5e2b4938880eac572a3f5290, State: Initialized, Role: FOLLOWER
I20260812 08:01:11.887220 32143 heartbeater.cc:499] Master 127.22.208.190:36717 was elected leader, sending a full tablet report...
I20260812 08:01:11.887458 32308 ts_tablet_manager.cc:1434] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:01:11.887688 32306 consensus_queue.cc:260] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.888355 32286 heartbeater.cc:499] Master 127.22.208.190:36717 was elected leader, sending a full tablet report...
I20260812 08:01:11.888685 32306 ts_tablet_manager.cc:1434] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260812 08:01:11.889834 32307 raft_consensus.cc:359] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.890147 32307 raft_consensus.cc:385] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:11.890354 32307 raft_consensus.cc:740] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37575efb04814de79c4ef62b1a472a69, State: Initialized, Role: FOLLOWER
I20260812 08:01:11.890918 32307 consensus_queue.cc:260] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:11.892128 31994 heartbeater.cc:499] Master 127.22.208.190:36717 was elected leader, sending a full tablet report...
I20260812 08:01:11.892175 32307 ts_tablet_manager.cc:1434] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
W20260812 08:01:11.921103 31995 tablet.cc:2406] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:01:12.065543 32287 tablet.cc:2406] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:01:12.111729 32144 tablet.cc:2406] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:01:12.147687 32313 raft_consensus.cc:493] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:01:12.147902 32313 raft_consensus.cc:515] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:12.149058 32313 leader_election.cc:290] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473), ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345)
I20260812 08:01:12.153882 31942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "049a7ef1624e4420952a5c0960ea2492" candidate_uuid: "dba6326e5e2b4938880eac572a3f5290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37575efb04814de79c4ef62b1a472a69" is_pre_election: true
I20260812 08:01:12.153965 32090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "049a7ef1624e4420952a5c0960ea2492" candidate_uuid: "dba6326e5e2b4938880eac572a3f5290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcf1211ef9942bf910d906bbbe02475" is_pre_election: true
I20260812 08:01:12.154240 31942 raft_consensus.cc:2468] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba6326e5e2b4938880eac572a3f5290 in term 0.
I20260812 08:01:12.154263 32090 raft_consensus.cc:2468] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba6326e5e2b4938880eac572a3f5290 in term 0.
I20260812 08:01:12.154817 32165 leader_election.cc:304] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba6326e5e2b4938880eac572a3f5290, ddcf1211ef9942bf910d906bbbe02475; no voters: 
I20260812 08:01:12.155136 32313 raft_consensus.cc:2804] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:01:12.155230 32313 raft_consensus.cc:493] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:01:12.155279 32313 raft_consensus.cc:3060] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:12.156353 32313 raft_consensus.cc:515] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:12.156834 32313 leader_election.cc:290] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [CANDIDATE]: Term 1 election: Requested vote from peers 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473), ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345)
I20260812 08:01:12.157327 32090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "049a7ef1624e4420952a5c0960ea2492" candidate_uuid: "dba6326e5e2b4938880eac572a3f5290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcf1211ef9942bf910d906bbbe02475"
I20260812 08:01:12.157436 31942 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "049a7ef1624e4420952a5c0960ea2492" candidate_uuid: "dba6326e5e2b4938880eac572a3f5290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37575efb04814de79c4ef62b1a472a69"
I20260812 08:01:12.157488 32090 raft_consensus.cc:3060] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:12.157591 31942 raft_consensus.cc:3060] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:12.159122 31942 raft_consensus.cc:2468] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba6326e5e2b4938880eac572a3f5290 in term 1.
I20260812 08:01:12.159332 32090 raft_consensus.cc:2468] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba6326e5e2b4938880eac572a3f5290 in term 1.
I20260812 08:01:12.159600 32164 leader_election.cc:304] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37575efb04814de79c4ef62b1a472a69, dba6326e5e2b4938880eac572a3f5290; no voters: 
I20260812 08:01:12.159850 32313 raft_consensus.cc:2804] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:12.160086 32313 raft_consensus.cc:697] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [term 1 LEADER]: Becoming Leader. State: Replica: dba6326e5e2b4938880eac572a3f5290, State: Running, Role: LEADER
I20260812 08:01:12.160459 32313 consensus_queue.cc:237] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } }
I20260812 08:01:12.164645 31780 catalog_manager.cc:5719] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba6326e5e2b4938880eac572a3f5290 (127.22.208.131). New cstate: current_term: 1 leader_uuid: "dba6326e5e2b4938880eac572a3f5290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba6326e5e2b4938880eac572a3f5290" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 } health_report { overall_health: UNKNOWN } } }
I20260812 08:01:12.232509 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:01:12.234186 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 37575efb04814de79c4ef62b1a472a69 to finish bootstrapping
I20260812 08:01:12.240489 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ddcf1211ef9942bf910d906bbbe02475 to finish bootstrapping
I20260812 08:01:12.246232 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dba6326e5e2b4938880eac572a3f5290 to finish bootstrapping
I20260812 08:01:12.257478 31988 generic_service.cc:175] {username='slave'} at 127.0.0.1:54206 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1000'
I20260812 08:01:12.259030 31988 generic_service.cc:175] {username='slave'} at 127.0.0.1:54206 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260812 08:01:12.264513 32134 generic_service.cc:175] {username='slave'} at 127.0.0.1:35722 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1000'
I20260812 08:01:12.265781 32134 generic_service.cc:175] {username='slave'} at 127.0.0.1:35722 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260812 08:01:12.270376 32280 generic_service.cc:175] {username='slave'} at 127.0.0.1:52716 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1000'
I20260812 08:01:12.271678 32280 generic_service.cc:175] {username='slave'} at 127.0.0.1:52716 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260812 08:01:12.282414 23362 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
W20260812 08:01:12.295127 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 769ms of latency in SegmentAllocator::Sync()
I20260812 08:01:12.296041 31942 raft_consensus.cc:1275] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Refusing update from remote peer dba6326e5e2b4938880eac572a3f5290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:12.296039 32090 raft_consensus.cc:1275] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Refusing update from remote peer dba6326e5e2b4938880eac572a3f5290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:12.296710 32313 consensus_queue.cc:1048] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddcf1211ef9942bf910d906bbbe02475" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:12.296926 32316 consensus_queue.cc:1048] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37575efb04814de79c4ef62b1a472a69" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 08:01:12.303238 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 884ms of latency in SegmentAllocator::Sync()
W20260812 08:01:12.303318 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1075ms of latency in SegmentAllocator::Sync()
W20260812 08:01:12.347587 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT)
W20260812 08:01:12.399004 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:12.398967 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:13.188102 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 3) took 890 ms (client timeout 50 ms). Trace:
I20260812 08:01:13.188118 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
I20260812 08:01:13.188823 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
W20260812 08:01:13.189312 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 4) took 840 ms (client timeout 50 ms). Trace:
I20260812 08:01:13.189381 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
W20260812 08:01:13.189445 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 5) took 448 ms (client timeout 50 ms). Trace:
W20260812 08:01:13.188373 32090 rpcz_store.cc:269] 0812 08:01:12.297443 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.297509 (+    66us) service_pool.cc:224] Handling call
0812 08:01:12.297912 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:12.298272 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:12.298278 (+     6us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0812 08:01:12.299085 (+   807us) write_op.cc:270] Start()
0812 08:01:12.299311 (+   226us) write_op.cc:276] Timestamp: P: 1786521672294133 usec, L: 0
0812 08:01:12.300694 (+  1383us) log.cc:844] Serialized 371 byte log entry
0812 08:01:12.300935 (+   241us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:12.300947 (+    12us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:12.302119 (+  1172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:13.187935 (+885816us) raft_consensus.cc:1685] finished
0812 08:01:13.187940 (+     5us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.188060 (+   120us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:12.300187 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:12.300352 (+   165us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:12.300371 (+    19us) write_op.cc:435] Acquired schema lock
0812 08:01:12.300374 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:12.300884 (+   510us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:12.300940 (+    56us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:12.300948 (+     8us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:01:12.301701 (+   753us) tablet.cc:701] Row locks acquired
0812 08:01:12.301703 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":42,"raft.run_wall_time_us":43,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":1448,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":37,"prepare.queue_time_us":243,"prepare.run_cpu_time_us":2020,"prepare.run_wall_time_us":2019,"thread_start_us":104,"threads_started":1}]]}
W20260812 08:01:13.189399 32089 rpcz_store.cc:269] 0812 08:01:12.348711 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.348800 (+    89us) service_pool.cc:224] Handling call
0812 08:01:13.188044 (+839244us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.188629 (+   585us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.188633 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.188644 (+    11us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.188648 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.188677 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.189252 (+   575us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:13.189294 (+    42us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1929599872}
W20260812 08:01:13.189541 32088 rpcz_store.cc:269] 0812 08:01:12.741415 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.741532 (+   117us) service_pool.cc:224] Handling call
0812 08:01:13.188749 (+447217us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.189253 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.189256 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.189266 (+    10us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.189269 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.189282 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.189407 (+   125us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:13.189432 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1027756800}
W20260812 08:01:13.189747 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 6) took 397 ms (client timeout 50 ms). Trace:
W20260812 08:01:13.189803 32087 rpcz_store.cc:269] 0812 08:01:12.792539 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.792674 (+   135us) service_pool.cc:224] Handling call
0812 08:01:13.189346 (+396672us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.189554 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.189558 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.189567 (+     9us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.189571 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.189583 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.189705 (+   122us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:13.189734 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":911538944}
I20260812 08:01:13.261577 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
W20260812 08:01:13.280699 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 939ms of latency in SegmentAllocator::Sync()
W20260812 08:01:13.285893 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1118ms of latency in SegmentAllocator::Sync()
W20260812 08:01:13.379056 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 3) took 1081 ms (client timeout 50 ms). Trace:
I20260812 08:01:13.379151 31940 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
I20260812 08:01:13.379631 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
W20260812 08:01:13.379901 31940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 4) took 1031 ms (client timeout 50 ms). Trace:
W20260812 08:01:13.379984 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 6) took 586 ms (client timeout 50 ms). Trace:
W20260812 08:01:13.379240 31942 rpcz_store.cc:269] 0812 08:01:12.297506 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.297553 (+    47us) service_pool.cc:224] Handling call
0812 08:01:12.297912 (+   359us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:12.298272 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:12.298278 (+     6us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0812 08:01:12.299085 (+   807us) write_op.cc:270] Start()
0812 08:01:12.299311 (+   226us) write_op.cc:276] Timestamp: P: 1786521672294133 usec, L: 0
0812 08:01:12.300694 (+  1383us) log.cc:844] Serialized 371 byte log entry
0812 08:01:12.300942 (+   248us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:12.300950 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:12.303455 (+  2505us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:13.378939 (+1075484us) raft_consensus.cc:1685] finished
0812 08:01:13.378943 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.379021 (+    78us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:12.300187 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:12.300365 (+   178us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:12.300377 (+    12us) write_op.cc:435] Acquired schema lock
0812 08:01:12.300379 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:12.300866 (+   487us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:12.300940 (+    74us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:12.300948 (+     8us) tablet.cc:685] Acquiring locks for 1 operations
0812 08:01:12.301701 (+   753us) tablet.cc:701] Row locks acquired
0812 08:01:12.301704 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"raft.queue_time_us":83,"raft.run_cpu_time_us":42,"raft.run_wall_time_us":43,"thread_start_us":115,"threads_started":1,"wal-append.queue_time_us":1551,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":52,"prepare.queue_time_us":243,"prepare.run_cpu_time_us":2020,"prepare.run_wall_time_us":2019,"thread_start_us":104,"threads_started":1}]]}
I20260812 08:01:13.379899 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2]   Dedup: 1.2->[]
W20260812 08:01:13.380137 31939 rpcz_store.cc:269] 0812 08:01:12.793084 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.793147 (+    63us) service_pool.cc:224] Handling call
0812 08:01:13.379572 (+586425us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.379767 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.379771 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.379781 (+    10us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.379786 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.379798 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.379935 (+   137us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:13.379968 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1348279552}
W20260812 08:01:13.380059 31940 rpcz_store.cc:269] 0812 08:01:12.348675 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.348754 (+    79us) service_pool.cc:224] Handling call
0812 08:01:13.379063 (+1030309us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.379458 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.379462 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.379482 (+    20us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.379486 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.379497 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.379843 (+   346us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:13.379886 (+    43us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2368851200}
W20260812 08:01:13.380468 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 5) took 638 ms (client timeout 50 ms). Trace:
I20260812 08:01:13.380512 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.2->[1.3-1.3]
W20260812 08:01:13.380522 31941 rpcz_store.cc:269] 0812 08:01:12.742165 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:12.742301 (+   136us) service_pool.cc:224] Handling call
0812 08:01:13.379861 (+637560us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:13.380275 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.380278 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:13.380287 (+     9us) raft_consensus.cc:1645] Marking committed up to 0
0812 08:01:13.380291 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.380301 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:13.380429 (+   128us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:13.380455 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1465856384}
W20260812 08:01:13.382162 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1121ms of latency in SegmentAllocator::Sync()
W20260812 08:01:13.384868 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:14.283002 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8580
I20260812 08:01:14.404546 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.404551 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 9) took 1120 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.404742 32087 rpcz_store.cc:269] 0812 08:01:13.283588 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.283652 (+    64us) service_pool.cc:224] Handling call
0812 08:01:13.283864 (+   212us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:13.284001 (+   137us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:13.284005 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:13.284156 (+   151us) write_op.cc:270] Start()
0812 08:01:13.284191 (+    35us) write_op.cc:276] Timestamp: P: 1786521673280072 usec, L: 0
0812 08:01:13.284526 (+   335us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:13.284573 (+    47us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:13.284580 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.284596 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:14.404395 (+1119799us) raft_consensus.cc:1685] finished
0812 08:01:14.404398 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.404520 (+   122us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:13.284361 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:13.284480 (+   119us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:13.284487 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:13.284491 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:13.285024 (+   533us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:13.285043 (+    19us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:13.285047 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:13.285469 (+   422us) tablet.cc:701] Row locks acquired
0812 08:01:13.285471 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":8,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":1260,"prepare.run_wall_time_us":1260}]]}
I20260812 08:01:14.404958 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.405064 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 10) took 1070 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.405138 32090 rpcz_store.cc:269] 0812 08:01:13.334799 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.334924 (+   125us) service_pool.cc:224] Handling call
0812 08:01:14.404501 (+1069577us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.404739 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.404777 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.404799 (+    22us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.404803 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.404817 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.404991 (+   174us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:14.405024 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2459266816}
I20260812 08:01:14.405265 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.405330 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 11) took 683 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.405544 32088 rpcz_store.cc:269] 0812 08:01:13.721560 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.721637 (+    77us) service_pool.cc:224] Handling call
0812 08:01:14.404923 (+683286us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.405116 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.405119 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.405129 (+    10us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.405133 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.405145 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.405282 (+   137us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:14.405316 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1570863872}
W20260812 08:01:14.405790 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 12) took 633 ms (client timeout 50 ms). Trace:
I20260812 08:01:14.405793 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.405897 32089 rpcz_store.cc:269] 0812 08:01:13.772387 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.772455 (+    68us) service_pool.cc:224] Handling call
0812 08:01:14.405234 (+632779us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.405529 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.405535 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.405545 (+    10us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.405612 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.405625 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.405748 (+   123us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:14.405777 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1454817664}
I20260812 08:01:14.406121 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.406190 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 13) took 303 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.406237 32086 rpcz_store.cc:269] 0812 08:01:14.102438 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.102547 (+   109us) service_pool.cc:224] Handling call
0812 08:01:14.405758 (+303211us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.405997 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.406000 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.406010 (+    10us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.406015 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.406030 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.406147 (+   117us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:14.406178 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":696687360}
W20260812 08:01:14.406414 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 14) took 252 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.406466 32085 rpcz_store.cc:269] 0812 08:01:14.153560 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.153698 (+   138us) service_pool.cc:224] Handling call
0812 08:01:14.406087 (+252389us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.406240 (+   153us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.406247 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.406255 (+     8us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.406259 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.406268 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.406378 (+   110us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:14.406401 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":579824896}
I20260812 08:01:14.503804 31940 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.503927 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 7) took 1177 ms (client timeout 50 ms). Trace:
I20260812 08:01:14.504191 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.503998 31938 rpcz_store.cc:269] 0812 08:01:13.326427 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.326532 (+   105us) service_pool.cc:224] Handling call
0812 08:01:13.380441 (+ 53909us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:13.380683 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:13.381090 (+   407us) log.cc:844] Serialized 14 byte log entry
0812 08:01:13.381490 (+   400us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:13.381643 (+   153us) write_op.cc:270] Start()
0812 08:01:13.381709 (+    66us) write_op.cc:276] Timestamp: P: 1786521673280072 usec, L: 0
0812 08:01:13.381921 (+   212us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:13.381981 (+    60us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:13.381986 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:13.382007 (+    21us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:14.503605 (+1121598us) raft_consensus.cc:1685] finished
0812 08:01:14.503608 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.503843 (+   235us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.503900 (+    57us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:13.381874 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:13.381996 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:13.382000 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:13.382001 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:13.382379 (+   378us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:13.382401 (+    22us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:13.382404 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:13.382918 (+   514us) tablet.cc:701] Row locks acquired
0812 08:01:13.382921 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"prepare.queue_time_us":304,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":141,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":307,"spinlock_wait_cycles":123293952,"thread_start_us":256,"threads_started":2,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":16,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":1046,"prepare.run_wall_time_us":1119,"row_lock_wait_count":1,"row_lock_wait_us":107}]]}
W20260812 08:01:14.504268 31940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 10) took 653 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.504369 31940 rpcz_store.cc:269] 0812 08:01:13.850448 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.850611 (+   163us) service_pool.cc:224] Handling call
0812 08:01:14.503745 (+653134us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:14.504010 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.504014 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.504030 (+    16us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.504035 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.504048 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.504214 (+   166us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.504253 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501479808}
I20260812 08:01:14.504549 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.504495 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 8) took 1127 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.504686 31937 rpcz_store.cc:269] 0812 08:01:13.377331 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.377478 (+   147us) service_pool.cc:224] Handling call
0812 08:01:14.504124 (+1126646us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:14.504313 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.504317 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.504329 (+    12us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.504336 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.504352 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.504463 (+   111us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.504486 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2590482816}
I20260812 08:01:14.504812 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.504856 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 12) took 202 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.504896 31942 rpcz_store.cc:269] 0812 08:01:14.302535 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.302667 (+   132us) service_pool.cc:224] Handling call
0812 08:01:14.504515 (+201848us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:14.504669 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.504674 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.504682 (+     8us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.504685 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.504696 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.504818 (+   122us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.504846 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":463593216}
I20260812 08:01:14.505026 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3]   Dedup: 1.3->[]
W20260812 08:01:14.505045 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 9) took 705 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.505208 31941 rpcz_store.cc:269] 0812 08:01:13.799508 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.799703 (+   195us) service_pool.cc:224] Handling call
0812 08:01:14.504786 (+705083us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:14.504896 (+   110us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.504899 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.504907 (+     8us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.504912 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.504920 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.505017 (+    97us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.505035 (+    18us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1621021568}
W20260812 08:01:14.505389 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 11) took 253 ms (client timeout 50 ms). Trace:
W20260812 08:01:14.505462 31939 rpcz_store.cc:269] 0812 08:01:14.251519 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.251629 (+   110us) service_pool.cc:224] Handling call
0812 08:01:14.504997 (+253368us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:14.505190 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 2
0812 08:01:14.505195 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:14.505205 (+    10us) raft_consensus.cc:1645] Marking committed up to 2
0812 08:01:14.505209 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.505223 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:14.505331 (+   108us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:14.505351 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582103936}
I20260812 08:01:14.539523 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20260812 08:01:14.543800 32370 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=1, attempt_no=0}) took 1266 ms. Trace:
I20260812 08:01:14.544034 32370 rpcz_store.cc:276] 0812 08:01:13.276919 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.277259 (+   340us) service_pool.cc:224] Handling call
0812 08:01:14.543775 (+1266516us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:13.278927 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:13.279044 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:13.279048 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:13.279050 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:13.279482 (+   432us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:13.279493 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:13.279497 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:13.280098 (+   601us) tablet.cc:701] Row locks acquired
0812 08:01:13.280100 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:13.280143 (+    43us) write_op.cc:270] Start()
0812 08:01:13.280178 (+    35us) write_op.cc:276] Timestamp: P: 1786521673280072 usec, L: 0
0812 08:01:13.280180 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:13.280371 (+   191us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:14.541162 (+1260791us) op_driver.cc:464] REPLICATION: finished
0812 08:01:14.541507 (+   345us) write_op.cc:301] APPLY: starting
0812 08:01:14.541530 (+    23us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:14.541629 (+    99us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:14.541647 (+    18us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:14.543327 (+  1680us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:14.543391 (+    64us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:14.543397 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:14.543545 (+   148us) log.cc:844] Serialized 319 byte log entry
0812 08:01:14.543621 (+    76us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:14.543656 (+    35us) write_op.cc:454] Released schema lock
0812 08:01:14.543666 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256,"num_ops":50,"prepare.queue_time_us":1047,"prepare.run_cpu_time_us":1603,"prepare.run_wall_time_us":2640,"raft.queue_time_us":1380,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":433,"replication_time_us":1260939,"thread_start_us":109,"threads_started":1}]]}
W20260812 08:01:14.545135 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1184ms of latency in SegmentAllocator::Sync()
W20260812 08:01:14.546419 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1037ms of latency in SegmentAllocator::Sync()
W20260812 08:01:14.647367 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:14.846695 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.4]   Dedup: 1.3->[1.4-1.4]
W20260812 08:01:14.847962 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1040ms of latency in SegmentAllocator::Sync()
W20260812 08:01:14.947540 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:01:15.545043 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce82c0
W20260812 08:01:15.583997 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 17) took 1038 ms (client timeout 50 ms). Trace:
I20260812 08:01:15.584038 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.584164 32085 rpcz_store.cc:269] 0812 08:01:14.545359 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.545449 (+    90us) service_pool.cc:224] Handling call
0812 08:01:14.545711 (+   262us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:14.545786 (+    75us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:14.545788 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:14.545913 (+   125us) write_op.cc:270] Start()
0812 08:01:14.545937 (+    24us) write_op.cc:276] Timestamp: P: 1786521674543864 usec, L: 0
0812 08:01:14.546182 (+   245us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:14.546249 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:14.546255 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.546267 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:15.583846 (+1037579us) raft_consensus.cc:1685] finished
0812 08:01:15.583849 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.583966 (+   117us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:14.546676 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:14.546775 (+    99us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:14.546779 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:14.546780 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:14.546971 (+   191us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:14.546979 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:14.546982 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:14.547144 (+   162us) tablet.cc:701] Row locks acquired
0812 08:01:14.547145 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":610,"prepare.run_cpu_time_us":537,"prepare.run_wall_time_us":537,"thread_start_us":69,"threads_started":1}]]}
I20260812 08:01:15.584360 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.584478 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 19) took 464 ms (client timeout 50 ms). Trace:
W20260812 08:01:15.584523 32089 rpcz_store.cc:269] 0812 08:01:15.120410 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.120507 (+    97us) service_pool.cc:224] Handling call
0812 08:01:15.583949 (+463442us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:15.584239 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:15.584245 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:15.584261 (+    16us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:15.584268 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.584285 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.584426 (+   141us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:15.584462 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065211264}
W20260812 08:01:15.584674 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 20) took 412 ms (client timeout 50 ms). Trace:
W20260812 08:01:15.584714 32088 rpcz_store.cc:269] 0812 08:01:15.171981 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.172064 (+    83us) service_pool.cc:224] Handling call
0812 08:01:15.584330 (+412266us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:15.584503 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:15.584506 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:15.584513 (+     7us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:15.584516 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.584528 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.584642 (+   114us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:15.584664 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947499904}
I20260812 08:01:15.584867 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.585141 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 18) took 988 ms (client timeout 50 ms). Trace:
I20260812 08:01:15.585106 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.585242 32086 rpcz_store.cc:269] 0812 08:01:14.596985 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.597059 (+    74us) service_pool.cc:224] Handling call
0812 08:01:15.584839 (+987780us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:15.584975 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:15.584978 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:15.584986 (+     8us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:15.584989 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.585001 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.585109 (+   108us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:15.585131 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2271230336}
W20260812 08:01:15.585539 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 21) took 76 ms (client timeout 50 ms). Trace:
I20260812 08:01:15.585593 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.585590 32090 rpcz_store.cc:269] 0812 08:01:15.508835 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.508936 (+   101us) service_pool.cc:224] Handling call
0812 08:01:15.585076 (+ 76140us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:15.585345 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:15.585351 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:15.585360 (+     9us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:15.585366 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.585382 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.585505 (+   123us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:15.585528 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174479744}
W20260812 08:01:15.734227 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1002ms of latency in SegmentAllocator::Sync()
I20260812 08:01:15.734314 32381 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=2, attempt_no=0}) took 2457 ms. Trace:
I20260812 08:01:15.734432 32381 rpcz_store.cc:276] 0812 08:01:13.277029 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.277908 (+   879us) service_pool.cc:224] Handling call
0812 08:01:15.734263 (+2456355us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:13.281598 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:13.281958 (+   360us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:13.281964 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:13.281966 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:13.282280 (+   314us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:13.282311 (+    31us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:13.282324 (+    13us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:14.543838 (+1261514us) lock_manager.cc:598] Waited 1261077us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:14.543854 (+    16us) tablet.cc:701] Row locks acquired
0812 08:01:14.543856 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:14.543912 (+    56us) write_op.cc:270] Start()
0812 08:01:14.543939 (+    27us) write_op.cc:276] Timestamp: P: 1786521674543864 usec, L: 0
0812 08:01:14.543941 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:14.544096 (+   155us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:15.729834 (+1185738us) op_driver.cc:464] REPLICATION: finished
0812 08:01:15.730237 (+   403us) write_op.cc:301] APPLY: starting
0812 08:01:15.730283 (+    46us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:15.730371 (+    88us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:15.730372 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:15.732079 (+  1707us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:15.732125 (+    46us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:15.732129 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:15.732235 (+   106us) log.cc:844] Serialized 319 byte log entry
0812 08:01:15.732323 (+    88us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:15.734154 (+  1831us) write_op.cc:454] Released schema lock
0812 08:01:15.734168 (+    14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":306,"num_ops":50,"prepare.queue_time_us":3349,"prepare.run_cpu_time_us":1882,"prepare.run_wall_time_us":1262594,"raft.queue_time_us":1479,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":203,"replication_time_us":1185827,"row_lock_wait_count":1,"row_lock_wait_us":1261077,"thread_start_us":136,"threads_started":1}]]}
W20260812 08:01:15.735291 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1105ms of latency in SegmentAllocator::Sync()
W20260812 08:01:15.836094 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:15.888692 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.4]   Dedup: 1.4->[]
W20260812 08:01:15.888787 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 13) took 1042 ms (client timeout 50 ms). Trace:
W20260812 08:01:15.888851 31939 rpcz_store.cc:269] 0812 08:01:14.846119 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.846335 (+   216us) service_pool.cc:224] Handling call
0812 08:01:14.846621 (+   286us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:14.846931 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:14.847171 (+   240us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:14.847311 (+   140us) write_op.cc:270] Start()
0812 08:01:14.847339 (+    28us) write_op.cc:276] Timestamp: P: 1786521674543864 usec, L: 0
0812 08:01:14.847657 (+   318us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:14.847705 (+    48us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:14.847711 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:14.847725 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:15.888537 (+1040812us) raft_consensus.cc:1685] finished
0812 08:01:15.888540 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.888762 (+   222us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:14.847565 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:14.847698 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:14.847702 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:14.847704 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:14.848015 (+   311us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:14.848026 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:14.848028 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:14.849114 (+  1086us) tablet.cc:701] Row locks acquired
0812 08:01:14.849116 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":1006,"prepare.run_wall_time_us":1715,"row_lock_wait_count":1,"row_lock_wait_us":807,"thread_start_us":80,"threads_started":1}]]}
W20260812 08:01:15.889094 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 15) took 619 ms (client timeout 50 ms). Trace:
I20260812 08:01:15.889046 31940 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.4->[1.5-1.5]
W20260812 08:01:15.889155 31942 rpcz_store.cc:269] 0812 08:01:15.270052 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.270216 (+   164us) service_pool.cc:224] Handling call
0812 08:01:15.888637 (+618421us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:15.888889 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:15.888892 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:15.888905 (+    13us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:15.888909 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.888922 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:15.889060 (+   138us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:15.889083 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421709184}
W20260812 08:01:15.890429 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 996ms of latency in SegmentAllocator::Sync()
W20260812 08:01:16.734293 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8b00
I20260812 08:01:16.840969 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.840969 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 24) took 1106 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.841212 32087 rpcz_store.cc:269] 0812 08:01:15.734261 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.734314 (+    53us) service_pool.cc:224] Handling call
0812 08:01:15.734477 (+   163us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:15.734560 (+    83us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:15.734563 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:15.734733 (+   170us) write_op.cc:270] Start()
0812 08:01:15.734755 (+    22us) write_op.cc:276] Timestamp: P: 1786521675732350 usec, L: 0
0812 08:01:15.735073 (+   318us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:15.735107 (+    34us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:15.735113 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.735124 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:16.840740 (+1105616us) raft_consensus.cc:1685] finished
0812 08:01:16.840742 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.840937 (+   195us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:15.734999 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:15.735103 (+   104us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:15.735107 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:15.735108 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:15.735446 (+   338us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:15.735457 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:15.735460 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:15.735701 (+   241us) tablet.cc:701] Row locks acquired
0812 08:01:15.735703 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":164,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":756,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:01:16.841421 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.841516 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 26) took 540 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.841601 32086 rpcz_store.cc:269] 0812 08:01:16.301379 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.301462 (+    83us) service_pool.cc:224] Handling call
0812 08:01:16.840910 (+539448us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.841249 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.841253 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.841264 (+    11us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.841269 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.841281 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.841472 (+   191us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:16.841503 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239937920}
W20260812 08:01:16.841836 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 25) took 1055 ms (client timeout 50 ms). Trace:
I20260812 08:01:16.841852 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.841926 32090 rpcz_store.cc:269] 0812 08:01:15.785999 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.786140 (+   141us) service_pool.cc:224] Handling call
0812 08:01:16.841369 (+1055229us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.841595 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.841600 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.841611 (+    11us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.841617 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.841634 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.841779 (+   145us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:16.841819 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2426265344}
W20260812 08:01:16.842365 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 28) took 73 ms (client timeout 50 ms). Trace:
I20260812 08:01:16.842371 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.842440 32089 rpcz_store.cc:269] 0812 08:01:16.768979 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.769086 (+   107us) service_pool.cc:224] Handling call
0812 08:01:16.841827 (+ 72741us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.841978 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.841981 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.841988 (+     7us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.841992 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.842002 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.842322 (+   320us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:16.842349 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166712576}
I20260812 08:01:16.842684 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.843034 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 27) took 490 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.843088 32088 rpcz_store.cc:269] 0812 08:01:16.352787 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.352930 (+   143us) service_pool.cc:224] Handling call
0812 08:01:16.842651 (+489721us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.842850 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.842853 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.842860 (+     7us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.842864 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.842889 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.843001 (+   112us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:16.843022 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125635328}
I20260812 08:01:16.846143 32391 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=3, attempt_no=0}) took 3569 ms. Trace:
W20260812 08:01:16.846518 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 906ms of latency in SegmentAllocator::Sync()
I20260812 08:01:16.846258 32391 rpcz_store.cc:276] 0812 08:01:13.277128 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.278266 (+  1138us) service_pool.cc:224] Handling call
0812 08:01:16.846087 (+3567821us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:14.544214 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:14.544336 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:14.544340 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:14.544341 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:14.544631 (+   290us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:14.544640 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:14.544645 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:15.732360 (+1187715us) lock_manager.cc:598] Waited 1187440us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:15.732373 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:15.732375 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:15.732424 (+    49us) write_op.cc:270] Start()
0812 08:01:15.732445 (+    21us) write_op.cc:276] Timestamp: P: 1786521675732350 usec, L: 0
0812 08:01:15.732447 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:15.732639 (+   192us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:16.843878 (+1111239us) op_driver.cc:464] REPLICATION: finished
0812 08:01:16.844298 (+   420us) write_op.cc:301] APPLY: starting
0812 08:01:16.844351 (+    53us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:16.844412 (+    61us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:16.844413 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:16.845737 (+  1324us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:16.845776 (+    39us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:16.845780 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:16.845876 (+    96us) log.cc:844] Serialized 319 byte log entry
0812 08:01:16.845959 (+    83us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:16.845992 (+    33us) write_op.cc:454] Released schema lock
0812 08:01:16.846000 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":276,"num_ops":50,"prepare.queue_time_us":1265631,"prepare.run_cpu_time_us":1306,"prepare.run_wall_time_us":1188519,"raft.queue_time_us":2388,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"replication_time_us":1111391,"row_lock_wait_count":1,"row_lock_wait_us":1187440,"thread_start_us":83,"threads_started":1}]]}
W20260812 08:01:16.849010 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1060ms of latency in SegmentAllocator::Sync()
I20260812 08:01:16.887171 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.4]   Dedup: 1.4->[]
W20260812 08:01:16.887274 31940 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 17) took 1022 ms (client timeout 50 ms). Trace:
I20260812 08:01:16.887508 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.887581 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 16) took 1566 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.887404 31940 rpcz_store.cc:269] 0812 08:01:15.864535 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.864699 (+   164us) service_pool.cc:224] Handling call
0812 08:01:15.888957 (+ 24258us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:15.889321 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:15.889622 (+   301us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:15.889785 (+   163us) write_op.cc:270] Start()
0812 08:01:15.889818 (+    33us) write_op.cc:276] Timestamp: P: 1786521675732350 usec, L: 0
0812 08:01:15.890155 (+   337us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:15.890221 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:15.890227 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:15.890240 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:16.887003 (+996763us) raft_consensus.cc:1685] finished
0812 08:01:16.887006 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.887189 (+   183us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.887250 (+    61us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:15.890110 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:15.890333 (+   223us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:15.890339 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:15.890343 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:15.890793 (+   450us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:15.890806 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:15.890809 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:15.891503 (+   694us) tablet.cc:701] Row locks acquired
0812 08:01:15.891505 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":55206784,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":1016,"prepare.run_wall_time_us":1450,"row_lock_wait_count":1,"row_lock_wait_us":510,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:01:16.887676 31937 rpcz_store.cc:269] 0812 08:01:15.321096 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.321254 (+   158us) service_pool.cc:224] Handling call
0812 08:01:16.887123 (+1565869us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:16.887360 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:16.887363 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.887373 (+    10us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:16.887376 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.887387 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.887536 (+   149us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.887568 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3600519552}
I20260812 08:01:16.887785 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.4]   Dedup: 1.4->[]
W20260812 08:01:16.887814 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 19) took 570 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.887868 31939 rpcz_store.cc:269] 0812 08:01:16.317498 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.317604 (+   106us) service_pool.cc:224] Handling call
0812 08:01:16.887474 (+569870us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:16.887633 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.887636 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.887645 (+     9us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.887648 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.887659 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.887775 (+   116us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.887803 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1309857024}
I20260812 08:01:16.887995 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.888044 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 14) took 1990 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.888084 31941 rpcz_store.cc:269] 0812 08:01:14.897304 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.897449 (+   145us) service_pool.cc:224] Handling call
0812 08:01:16.887764 (+1990315us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:16.887875 (+   111us) raft_consensus.cc:1534] Early marking committed up to index 3
0812 08:01:16.887880 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.887888 (+     8us) raft_consensus.cc:1645] Marking committed up to 3
0812 08:01:16.887891 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.887902 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.888006 (+   104us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.888033 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4576615808}
W20260812 08:01:16.888217 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 18) took 972 ms (client timeout 50 ms). Trace:
I20260812 08:01:16.888263 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.888260 31942 rpcz_store.cc:269] 0812 08:01:15.915969 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.916066 (+    97us) service_pool.cc:224] Handling call
0812 08:01:16.887966 (+971900us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:16.888073 (+   107us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.888075 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.888082 (+     7us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.888086 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.888096 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.888188 (+    92us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.888207 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2234576640}
W20260812 08:01:16.888770 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 20) took 520 ms (client timeout 50 ms). Trace:
I20260812 08:01:16.888805 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.888868 31938 rpcz_store.cc:269] 0812 08:01:16.368474 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.368684 (+   210us) service_pool.cc:224] Handling call
0812 08:01:16.888228 (+519544us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:16.888585 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.888589 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.888599 (+    10us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.888603 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.888616 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.888728 (+   112us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.888756 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1194216320}
I20260812 08:01:16.889138 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5]   Dedup: 1.5->[]
W20260812 08:01:16.889433 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 21) took 71 ms (client timeout 50 ms). Trace:
W20260812 08:01:16.889497 31936 rpcz_store.cc:269] 0812 08:01:16.818012 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.818156 (+   144us) service_pool.cc:224] Handling call
0812 08:01:16.889092 (+ 70936us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:16.889266 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 4
0812 08:01:16.889269 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:16.889277 (+     8us) raft_consensus.cc:1645] Marking committed up to 4
0812 08:01:16.889284 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.889297 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:16.889397 (+   100us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:16.889420 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162561792}
W20260812 08:01:16.892325 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1098ms of latency in SegmentAllocator::Sync()
W20260812 08:01:16.949903 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:16.992110 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:17.479664 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 22 similar messages]
W20260812 08:01:17.847446 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8dc0
I20260812 08:01:17.909639 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.909639 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 31) took 1061 ms (client timeout 50 ms). Trace:
I20260812 08:01:17.910033 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.910096 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 33) took 480 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.909870 32088 rpcz_store.cc:269] 0812 08:01:16.847718 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.847796 (+    78us) service_pool.cc:224] Handling call
0812 08:01:16.847984 (+   188us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.848070 (+    86us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:16.848076 (+     6us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:16.848210 (+   134us) write_op.cc:270] Start()
0812 08:01:16.848244 (+    34us) write_op.cc:276] Timestamp: P: 1786521676846037 usec, L: 0
0812 08:01:16.848566 (+   322us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:16.848609 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:16.848615 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.848628 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:17.909464 (+1060836us) raft_consensus.cc:1685] finished
0812 08:01:17.909468 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.909603 (+   135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:16.848498 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:16.848612 (+   114us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:16.848616 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:16.848618 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:16.849191 (+   573us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:16.849207 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:16.849213 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:16.849409 (+   196us) tablet.cc:701] Row locks acquired
0812 08:01:16.849410 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":960,"prepare.run_wall_time_us":959,"thread_start_us":99,"threads_started":1}]]}
W20260812 08:01:17.910187 32089 rpcz_store.cc:269] 0812 08:01:17.429362 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.429467 (+   105us) service_pool.cc:224] Handling call
0812 08:01:17.909583 (+480116us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.909854 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.909858 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.909872 (+    14us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.909878 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.909895 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.910046 (+   151us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:17.910084 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1103533824}
I20260812 08:01:17.910385 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.910476 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 34) took 429 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.910530 32090 rpcz_store.cc:269] 0812 08:01:17.480722 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.480819 (+    97us) service_pool.cc:224] Handling call
0812 08:01:17.909985 (+429166us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.910242 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.910246 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.910255 (+     9us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.910258 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.910280 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.910427 (+   147us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:17.910462 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986416512}
W20260812 08:01:17.910665 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 32) took 1010 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.910715 32085 rpcz_store.cc:269] 0812 08:01:16.899988 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.900353 (+   365us) service_pool.cc:224] Handling call
0812 08:01:17.910352 (+1009999us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.910478 (+   126us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.910482 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.910491 (+     9us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.910495 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.910504 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.910620 (+   116us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:17.910653 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2322328320}
W20260812 08:01:17.991186 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 23) took 1100 ms (client timeout 50 ms). Trace:
I20260812 08:01:17.991282 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20260812 08:01:17.991624 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.991320 31936 rpcz_store.cc:269] 0812 08:01:16.890939 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.891029 (+    90us) service_pool.cc:224] Handling call
0812 08:01:16.891191 (+   162us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:16.891276 (+    85us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:16.891549 (+   273us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:16.891717 (+   168us) write_op.cc:270] Start()
0812 08:01:16.891759 (+    42us) write_op.cc:276] Timestamp: P: 1786521676846037 usec, L: 0
0812 08:01:16.892098 (+   339us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:16.892142 (+    44us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:16.892149 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:16.892177 (+    28us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:17.991062 (+1098885us) raft_consensus.cc:1685] finished
0812 08:01:17.991065 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.991159 (+    94us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:16.892009 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:16.892468 (+   459us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:16.892474 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:16.892478 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:16.892840 (+   362us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:16.892856 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:16.892862 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:16.893700 (+   838us) tablet.cc:701] Row locks acquired
0812 08:01:16.893703 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":989,"prepare.run_wall_time_us":1745,"row_lock_wait_count":1,"row_lock_wait_us":518,"thread_start_us":92,"threads_started":1}]]}
W20260812 08:01:17.991744 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 25) took 556 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.991794 31938 rpcz_store.cc:269] 0812 08:01:17.435183 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.435268 (+    85us) service_pool.cc:224] Handling call
0812 08:01:17.991218 (+555950us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.991499 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.991501 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.991512 (+    11us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.991515 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.991526 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.991693 (+   167us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:17.991730 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1277936384}
I20260812 08:01:17.991945 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.991954 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 26) took 505 ms (client timeout 50 ms). Trace:
I20260812 08:01:17.992383 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.992324 31942 rpcz_store.cc:269] 0812 08:01:17.486539 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.486804 (+   265us) service_pool.cc:224] Handling call
0812 08:01:17.991595 (+504791us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.991751 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.991757 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.991767 (+    10us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.991772 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.991782 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.991920 (+   138us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:17.991942 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1160474880}
I20260812 08:01:17.992673 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
W20260812 08:01:17.993031 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 24) took 1051 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.993054 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 27) took 64 ms (client timeout 50 ms). Trace:
W20260812 08:01:17.993147 31941 rpcz_store.cc:269] 0812 08:01:17.928380 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.928485 (+   105us) service_pool.cc:224] Handling call
0812 08:01:17.992640 (+ 64155us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.992818 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.992822 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.992832 (+    10us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.992841 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.992859 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.992991 (+   132us) spinlock_profiling.cc:243] Waited 63.9 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:17.993038 (+    47us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147060608}
W20260812 08:01:17.993127 31935 rpcz_store.cc:269] 0812 08:01:16.942103 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.942310 (+   207us) service_pool.cc:224] Handling call
0812 08:01:17.991909 (+1049599us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.992291 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 5
0812 08:01:17.992294 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:17.992303 (+     9us) raft_consensus.cc:1645] Marking committed up to 5
0812 08:01:17.992306 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.992321 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:17.992988 (+   667us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:17.993018 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2413320576}
I20260812 08:01:17.997392 32404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=4, attempt_no=0}) took 4720 ms. Trace:
I20260812 08:01:17.997512 32404 rpcz_store.cc:276] 0812 08:01:13.277202 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.278635 (+  1433us) service_pool.cc:224] Handling call
0812 08:01:17.997368 (+4718733us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:15.732781 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:15.732935 (+   154us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:15.732941 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:15.732945 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:15.733529 (+   584us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:15.733541 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:15.733560 (+    19us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:16.846020 (+1112460us) lock_manager.cc:598] Waited 1111895us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:16.846036 (+    16us) tablet.cc:701] Row locks acquired
0812 08:01:16.846038 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:16.846119 (+    81us) write_op.cc:270] Start()
0812 08:01:16.846151 (+    32us) write_op.cc:276] Timestamp: P: 1786521676846037 usec, L: 0
0812 08:01:16.846153 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:16.846345 (+   192us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:17.993571 (+1147226us) op_driver.cc:464] REPLICATION: finished
0812 08:01:17.993785 (+   214us) write_op.cc:301] APPLY: starting
0812 08:01:17.993860 (+    75us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:17.993929 (+    69us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:17.993933 (+     4us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:17.996840 (+  2907us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:17.996886 (+    46us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:17.996890 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:17.997160 (+   270us) log.cc:844] Serialized 319 byte log entry
0812 08:01:17.997256 (+    96us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:17.997286 (+    30us) write_op.cc:454] Released schema lock
0812 08:01:17.997294 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"num_ops":50,"prepare.queue_time_us":2453899,"prepare.run_cpu_time_us":2104,"prepare.run_wall_time_us":1113675,"raft.queue_time_us":462,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":152,"replication_time_us":1147382,"row_lock_wait_count":1,"row_lock_wait_us":1111895,"spinlock_wait_cycles":35712,"thread_start_us":71,"threads_started":1}]]}
W20260812 08:01:17.998375 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 872ms of latency in SegmentAllocator::Sync()
W20260812 08:01:17.999914 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 879ms of latency in SegmentAllocator::Sync()
I20260812 08:01:18.027421 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7]   Dedup: 1.6->[1.7-1.7]
W20260812 08:01:18.028780 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1011ms of latency in SegmentAllocator::Sync()
W20260812 08:01:18.100669 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:18.879446 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20260812 08:01:18.879462 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 30) took 880 ms (client timeout 50 ms). Trace:
I20260812 08:01:18.879933 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20260812 08:01:18.879859 31935 rpcz_store.cc:269] 0812 08:01:17.998911 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:17.998982 (+    71us) service_pool.cc:224] Handling call
0812 08:01:17.999141 (+   159us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:17.999222 (+    81us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:17.999226 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:17.999380 (+   154us) write_op.cc:270] Start()
0812 08:01:17.999404 (+    24us) write_op.cc:276] Timestamp: P: 1786521677997466 usec, L: 0
0812 08:01:17.999676 (+   272us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:17.999741 (+    65us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:17.999747 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:17.999760 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:18.879335 (+879575us) raft_consensus.cc:1685] finished
0812 08:01:18.879337 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:18.879437 (+   100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:17.999681 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:17.999790 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:17.999794 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:17.999795 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:18.000097 (+   302us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:18.000109 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:18.000112 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:18.000498 (+   386us) tablet.cc:701] Row locks acquired
0812 08:01:18.000501 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":196,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":878,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:01:18.880277 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 31) took 829 ms (client timeout 50 ms). Trace:
I20260812 08:01:18.880309 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20260812 08:01:18.880335 31941 rpcz_store.cc:269] 0812 08:01:18.050401 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.050483 (+    82us) service_pool.cc:224] Handling call
0812 08:01:18.879902 (+829419us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:18.880090 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:18.880093 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:18.880101 (+     8us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:18.880105 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:18.880116 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:18.880227 (+   111us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:18.880263 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1906872704}
I20260812 08:01:18.880581 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20260812 08:01:18.880599 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 33) took 377 ms (client timeout 50 ms). Trace:
W20260812 08:01:18.880772 31942 rpcz_store.cc:269] 0812 08:01:18.503691 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.503817 (+   126us) service_pool.cc:224] Handling call
0812 08:01:18.880287 (+376470us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:18.880398 (+   111us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:18.880401 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:18.880407 (+     6us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:18.880410 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:18.880418 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:18.880562 (+   144us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:18.880586 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":865286912}
W20260812 08:01:18.881054 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 32) took 428 ms (client timeout 50 ms). Trace:
W20260812 08:01:18.881119 31939 rpcz_store.cc:269] 0812 08:01:18.452166 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.452258 (+    92us) service_pool.cc:224] Handling call
0812 08:01:18.880551 (+428293us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:18.880762 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:18.880765 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:18.880776 (+    11us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:18.880781 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:18.880791 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:18.880917 (+   126us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:18.881035 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984389888}
I20260812 08:01:18.883316 32411 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=5, attempt_no=0}) took 5600 ms. Trace:
I20260812 08:01:18.883411 32411 rpcz_store.cc:276] 0812 08:01:13.283225 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.284674 (+  1449us) service_pool.cc:224] Handling call
0812 08:01:18.883245 (+5598571us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:16.846483 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:16.846643 (+   160us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:16.846647 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:16.846648 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:16.847001 (+   353us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:16.847010 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:16.847014 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:17.997401 (+1150387us) lock_manager.cc:598] Waited 1150089us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:17.997414 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:17.997415 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:17.997533 (+   118us) write_op.cc:270] Start()
0812 08:01:17.997621 (+    88us) write_op.cc:276] Timestamp: P: 1786521677997466 usec, L: 0
0812 08:01:17.997626 (+     5us) op_driver.cc:348] REPLICATION: starting
0812 08:01:17.998091 (+   465us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:18.880619 (+882528us) op_driver.cc:464] REPLICATION: finished
0812 08:01:18.881103 (+   484us) write_op.cc:301] APPLY: starting
0812 08:01:18.881147 (+    44us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:18.881213 (+    66us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:18.881214 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:18.882528 (+  1314us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:18.882867 (+   339us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:18.882874 (+     7us) write_op.cc:312] APPLY: finished
0812 08:01:18.883041 (+   167us) log.cc:844] Serialized 319 byte log entry
0812 08:01:18.883112 (+    71us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:18.883142 (+    30us) write_op.cc:454] Released schema lock
0812 08:01:18.883158 (+    16us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":106624,"child_traces":[["op",{"apply.queue_time_us":413,"num_ops":50,"prepare.queue_time_us":3561455,"prepare.run_cpu_time_us":2100,"prepare.run_wall_time_us":1151829,"raft.queue_time_us":343,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"replication_time_us":882951,"row_lock_wait_count":1,"row_lock_wait_us":1150089,"thread_start_us":101,"threads_started":1}]]}
W20260812 08:01:18.883685 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 883ms of latency in SegmentAllocator::Sync()
W20260812 08:01:18.886340 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 942ms of latency in SegmentAllocator::Sync()
W20260812 08:01:18.986608 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:19.040746 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 35) took 1013 ms (client timeout 50 ms). Trace:
I20260812 08:01:19.040782 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7]   Dedup: 1.7->[]
W20260812 08:01:19.040896 32085 rpcz_store.cc:269] 0812 08:01:18.027009 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.027143 (+   134us) service_pool.cc:224] Handling call
0812 08:01:18.027345 (+   202us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:18.027622 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:18.027895 (+   273us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:18.028053 (+   158us) write_op.cc:270] Start()
0812 08:01:18.028082 (+    29us) write_op.cc:276] Timestamp: P: 1786521677997466 usec, L: 0
0812 08:01:18.028443 (+   361us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:18.028494 (+    51us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:18.028501 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:18.028518 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:19.040523 (+1012005us) raft_consensus.cc:1685] finished
0812 08:01:19.040526 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.040714 (+   188us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:18.028390 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:18.028542 (+   152us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:18.028546 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:18.028548 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:18.028895 (+   347us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:18.028947 (+    52us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:18.028954 (+     7us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:18.029945 (+   991us) tablet.cc:701] Row locks acquired
0812 08:01:18.029947 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":21,"prepare.queue_time_us":202,"prepare.run_cpu_time_us":915,"prepare.run_wall_time_us":1616,"row_lock_wait_count":1,"row_lock_wait_us":771,"thread_start_us":93,"threads_started":1}]]}
I20260812 08:01:19.041275 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7]   Dedup: 1.7->[]
W20260812 08:01:19.041394 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 37) took 482 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.041455 32088 rpcz_store.cc:269] 0812 08:01:18.558963 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.559077 (+   114us) service_pool.cc:224] Handling call
0812 08:01:19.040729 (+481652us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:19.041006 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:19.041018 (+    12us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.041111 (+    93us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:19.041120 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.041137 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.041338 (+   201us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.041378 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1107123456}
W20260812 08:01:19.041592 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 38) took 431 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.041675 32089 rpcz_store.cc:269] 0812 08:01:18.610018 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.610085 (+    67us) service_pool.cc:224] Handling call
0812 08:01:19.041236 (+431151us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:19.041416 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:19.041420 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.041428 (+     8us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:19.041433 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.041446 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.041559 (+   113us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.041583 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":991072640}
I20260812 08:01:19.041599 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7]   Dedup: 1.7->[]
W20260812 08:01:19.042232 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 36) took 964 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.042309 32090 rpcz_store.cc:269] 0812 08:01:18.077913 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.078022 (+   109us) service_pool.cc:224] Handling call
0812 08:01:19.041562 (+963540us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:19.042053 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 6
0812 08:01:19.042059 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.042071 (+    12us) raft_consensus.cc:1645] Marking committed up to 6
0812 08:01:19.042076 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.042088 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.042193 (+   105us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.042219 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2215461120}
I20260812 08:01:19.068797 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.8]   Dedup: 1.7->[1.8-1.8]
W20260812 08:01:19.070161 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 908ms of latency in SegmentAllocator::Sync()
W20260812 08:01:19.548238 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
I20260812 08:01:19.829035 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.829037 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 36) took 943 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.829296 31939 rpcz_store.cc:269] 0812 08:01:18.885270 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.885361 (+    91us) service_pool.cc:224] Handling call
0812 08:01:18.885506 (+   145us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:18.885575 (+    69us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:18.885577 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:18.885708 (+   131us) write_op.cc:270] Start()
0812 08:01:18.885731 (+    23us) write_op.cc:276] Timestamp: P: 1786521678883206 usec, L: 0
0812 08:01:18.886059 (+   328us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:18.886101 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:18.886107 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:18.886130 (+    23us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:19.828869 (+942739us) raft_consensus.cc:1685] finished
0812 08:01:19.828872 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.829014 (+   142us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:18.885987 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:18.886103 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:18.886106 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:18.886108 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:18.886403 (+   295us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:18.886413 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:18.886416 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:18.886712 (+   296us) tablet.cc:701] Row locks acquired
0812 08:01:18.886714 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":181,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":774,"thread_start_us":106,"threads_started":1}]]}
I20260812 08:01:19.829559 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.829583 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 39) took 496 ms (client timeout 50 ms). Trace:
I20260812 08:01:19.829818 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.829927 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 38) took 548 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.829811 31935 rpcz_store.cc:269] 0812 08:01:19.333007 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.333143 (+   136us) service_pool.cc:224] Handling call
0812 08:01:19.828982 (+495839us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:19.829325 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.829329 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.829341 (+    12us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.829345 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.829358 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.829539 (+   181us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:19.829569 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139842816}
W20260812 08:01:19.830040 31941 rpcz_store.cc:269] 0812 08:01:19.281622 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.281724 (+   102us) service_pool.cc:224] Handling call
0812 08:01:19.829519 (+547795us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:19.829712 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.829715 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.829725 (+    10us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.829732 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.829748 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.829881 (+   133us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:19.829914 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1259338624}
W20260812 08:01:19.830178 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 37) took 893 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.830510 31942 rpcz_store.cc:269] 0812 08:01:18.936420 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.936519 (+    99us) service_pool.cc:224] Handling call
0812 08:01:19.829792 (+893273us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:19.829991 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.829997 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.830006 (+     9us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.830011 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.830022 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.830137 (+   115us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:19.830165 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053768064}
W20260812 08:01:19.884642 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9080
I20260812 08:01:19.904927 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.908607 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 991ms of latency in SegmentAllocator::Sync()
I20260812 08:01:19.909232 32417 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=7, attempt_no=0}) took 6624 ms. Trace:
I20260812 08:01:19.909327 32417 rpcz_store.cc:276] 0812 08:01:13.284652 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.284704 (+    52us) service_pool.cc:224] Handling call
0812 08:01:19.909198 (+6624494us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:17.998362 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:17.998477 (+   115us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:17.998480 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:17.998481 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:17.998774 (+   293us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:17.998782 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:17.998785 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:18.883227 (+884442us) lock_manager.cc:598] Waited 884175us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:18.883246 (+    19us) tablet.cc:701] Row locks acquired
0812 08:01:18.883248 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:18.883293 (+    45us) write_op.cc:270] Start()
0812 08:01:18.883319 (+    26us) write_op.cc:276] Timestamp: P: 1786521678883206 usec, L: 0
0812 08:01:18.883321 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:18.883475 (+   154us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:19.906135 (+1022660us) op_driver.cc:464] REPLICATION: finished
0812 08:01:19.906419 (+   284us) write_op.cc:301] APPLY: starting
0812 08:01:19.906457 (+    38us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:19.906525 (+    68us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:19.906527 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:19.907905 (+  1378us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:19.907948 (+    43us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:19.907952 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:19.908054 (+   102us) log.cc:844] Serialized 319 byte log entry
0812 08:01:19.908137 (+    83us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:19.909105 (+   968us) write_op.cc:454] Released schema lock
0812 08:01:19.909115 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":13056,"child_traces":[["op",{"apply.queue_time_us":215,"mutex_wait_us":21,"num_ops":50,"prepare.queue_time_us":4713326,"prepare.run_cpu_time_us":1053,"prepare.run_wall_time_us":885210,"raft.queue_time_us":2040,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":255,"replication_time_us":1022775,"row_lock_wait_count":1,"row_lock_wait_us":884175,"thread_start_us":131,"threads_started":1}]]}
W20260812 08:01:19.910036 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1204ms of latency in SegmentAllocator::Sync()
I20260812 08:01:19.978762 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.978766 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 39) took 910 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.978966 32090 rpcz_store.cc:269] 0812 08:01:19.068089 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.068245 (+   156us) service_pool.cc:224] Handling call
0812 08:01:19.068700 (+   455us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:19.069019 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.069264 (+   245us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:19.069410 (+   146us) write_op.cc:270] Start()
0812 08:01:19.069440 (+    30us) write_op.cc:276] Timestamp: P: 1786521678883206 usec, L: 0
0812 08:01:19.069834 (+   394us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:19.069886 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.069893 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.069937 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:19.978603 (+908666us) raft_consensus.cc:1685] finished
0812 08:01:19.978606 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.978742 (+   136us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:19.069730 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:19.069869 (+   139us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:19.069873 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:19.069876 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:19.070224 (+   348us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:19.070236 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:19.070238 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:19.071256 (+  1018us) tablet.cc:701] Row locks acquired
0812 08:01:19.071258 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":199,"prepare.run_cpu_time_us":1111,"prepare.run_wall_time_us":1585,"row_lock_wait_count":1,"row_lock_wait_us":573,"thread_start_us":116,"threads_started":1}]]}
I20260812 08:01:19.979197 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.979197 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 42) took 481 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.979369 32085 rpcz_store.cc:269] 0812 08:01:19.498149 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.498283 (+   134us) service_pool.cc:224] Handling call
0812 08:01:19.978712 (+480429us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:19.978989 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.978992 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.979003 (+    11us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.979007 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.979024 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.979149 (+   125us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.979175 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1104320256}
I20260812 08:01:19.979560 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.8]   Dedup: 1.8->[]
W20260812 08:01:19.979709 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 41) took 532 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.979784 32088 rpcz_store.cc:269] 0812 08:01:19.446869 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.447222 (+   353us) service_pool.cc:224] Handling call
0812 08:01:19.979159 (+531937us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:19.979382 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.979388 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.979401 (+    13us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.979410 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.979452 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.979648 (+   196us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.979690 (+    42us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1222781184}
W20260812 08:01:19.979956 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 40) took 860 ms (client timeout 50 ms). Trace:
W20260812 08:01:19.980019 32089 rpcz_store.cc:269] 0812 08:01:19.119395 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.119479 (+    84us) service_pool.cc:224] Handling call
0812 08:01:19.979533 (+860054us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:19.979648 (+   115us) raft_consensus.cc:1534] Early marking committed up to index 7
0812 08:01:19.979654 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:19.979664 (+    10us) raft_consensus.cc:1645] Marking committed up to 7
0812 08:01:19.979670 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.979758 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:19.979895 (+   137us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:19.979939 (+    44us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1977391488}
I20260812 08:01:20.005097 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.8->[1.9-1.9]
W20260812 08:01:20.006374 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1164ms of latency in SegmentAllocator::Sync()
W20260812 08:01:20.010681 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:20.909386 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8000
W20260812 08:01:21.114719 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 42) took 1205 ms (client timeout 50 ms). Trace:
I20260812 08:01:21.114850 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.114873 31942 rpcz_store.cc:269] 0812 08:01:19.908990 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.909074 (+    84us) service_pool.cc:224] Handling call
0812 08:01:19.909229 (+   155us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:19.909313 (+    84us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:19.909316 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:19.909456 (+   140us) write_op.cc:270] Start()
0812 08:01:19.909476 (+    20us) write_op.cc:276] Timestamp: P: 1786521679908151 usec, L: 0
0812 08:01:19.909779 (+   303us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:19.909800 (+    21us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:19.909805 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:19.909817 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:21.114552 (+1204735us) raft_consensus.cc:1685] finished
0812 08:01:21.114554 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.114687 (+   133us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:19.909709 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:19.909810 (+   101us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:19.909813 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:19.909815 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:19.910151 (+   336us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:19.910161 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:19.910164 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:19.910430 (+   266us) tablet.cc:701] Row locks acquired
0812 08:01:19.910432 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":771,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:01:21.115231 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.115240 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 43) took 1155 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.115429 31941 rpcz_store.cc:269] 0812 08:01:19.960179 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.960326 (+   147us) service_pool.cc:224] Handling call
0812 08:01:21.114794 (+1154468us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.115056 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.115062 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.115078 (+    16us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.115083 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.115102 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.115197 (+    95us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:21.115228 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2654518144}
I20260812 08:01:21.115577 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.115657 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 44) took 757 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.115700 31935 rpcz_store.cc:269] 0812 08:01:20.358366 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.358477 (+   111us) service_pool.cc:224] Handling call
0812 08:01:21.115175 (+756698us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.115448 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.115451 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.115459 (+     8us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.115464 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.115480 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.115604 (+   124us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:21.115645 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1739566848}
W20260812 08:01:21.115824 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 45) took 706 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.115862 31939 rpcz_store.cc:269] 0812 08:01:20.409768 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.409879 (+   111us) service_pool.cc:224] Handling call
0812 08:01:21.115548 (+705669us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.115672 (+   124us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.115676 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.115683 (+     7us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.115686 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.115695 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.115794 (+    99us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:21.115814 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622339968}
I20260812 08:01:21.115990 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20260812 08:01:21.116302 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.116374 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 46) took 232 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.116417 31938 rpcz_store.cc:269] 0812 08:01:20.884205 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.884271 (+    66us) service_pool.cc:224] Handling call
0812 08:01:21.115968 (+231697us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.116088 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.116091 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.116098 (+     7us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.116102 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.116164 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.116328 (+   164us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:21.116359 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532386176}
W20260812 08:01:21.116562 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 47) took 181 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.116603 31936 rpcz_store.cc:269] 0812 08:01:20.935594 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.935768 (+   174us) service_pool.cc:224] Handling call
0812 08:01:21.116265 (+180497us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.116416 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.116419 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.116429 (+    10us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.116433 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.116442 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.116532 (+    90us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:21.116552 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414628608}
W20260812 08:01:21.171259 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 43) took 1166 ms (client timeout 50 ms). Trace:
I20260812 08:01:21.171259 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.171473 32089 rpcz_store.cc:269] 0812 08:01:20.004460 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.004550 (+    90us) service_pool.cc:224] Handling call
0812 08:01:20.005018 (+   468us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:20.005278 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:20.005515 (+   237us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:20.005643 (+   128us) write_op.cc:270] Start()
0812 08:01:20.005670 (+    27us) write_op.cc:276] Timestamp: P: 1786521679908151 usec, L: 0
0812 08:01:20.006003 (+   333us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:20.006050 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:20.006056 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:20.006069 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:21.171060 (+1164991us) raft_consensus.cc:1685] finished
0812 08:01:21.171063 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.171225 (+   162us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:20.005899 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:20.006025 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:20.006030 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:20.006032 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:20.006299 (+   267us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:20.006310 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:20.006312 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:20.007755 (+  1443us) tablet.cc:701] Row locks acquired
0812 08:01:20.007759 (+     4us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":1929,"row_lock_wait_count":1,"row_lock_wait_us":1185,"thread_start_us":89,"threads_started":1}]]}
I20260812 08:01:21.171700 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.172003 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 45) took 670 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.172071 32085 rpcz_store.cc:269] 0812 08:01:20.501141 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.501265 (+   124us) service_pool.cc:224] Handling call
0812 08:01:21.171191 (+669926us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:21.171568 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.171572 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.171585 (+    13us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.171590 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.171605 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.171931 (+   326us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:21.171985 (+    54us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540089728}
I20260812 08:01:21.172159 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.172243 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 48) took 223 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.172287 32087 rpcz_store.cc:269] 0812 08:01:20.949249 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.949387 (+   138us) service_pool.cc:224] Handling call
0812 08:01:21.171650 (+222263us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:21.172018 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.172020 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.172028 (+     8us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.172031 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.172040 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.172195 (+   155us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:21.172229 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510573952}
W20260812 08:01:21.172429 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 44) took 1117 ms (client timeout 50 ms). Trace:
I20260812 08:01:21.172516 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.172478 32088 rpcz_store.cc:269] 0812 08:01:20.055421 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.055544 (+   123us) service_pool.cc:224] Handling call
0812 08:01:21.172126 (+1116582us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:21.172258 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.172272 (+    14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.172279 (+     7us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.172283 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.172296 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.172396 (+   100us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:21.172420 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2567188736}
I20260812 08:01:21.172806 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.9]   Dedup: 1.9->[]
W20260812 08:01:21.172916 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 47) took 274 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.172986 32086 rpcz_store.cc:269] 0812 08:01:20.898309 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.898422 (+   113us) service_pool.cc:224] Handling call
0812 08:01:21.172493 (+274071us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:21.172623 (+   130us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.172628 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.172638 (+    10us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.172645 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.172658 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.172848 (+   190us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:21.172901 (+    53us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":629487360}
W20260812 08:01:21.173081 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 46) took 620 ms (client timeout 50 ms). Trace:
W20260812 08:01:21.173130 32090 rpcz_store.cc:269] 0812 08:01:20.552195 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:20.552280 (+    85us) service_pool.cc:224] Handling call
0812 08:01:21.172766 (+620486us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:21.172913 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 8
0812 08:01:21.172915 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:21.172923 (+     8us) raft_consensus.cc:1645] Marking committed up to 8
0812 08:01:21.172930 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.172942 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:21.173050 (+   108us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:21.173072 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1426420096}
I20260812 08:01:21.311271 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20260812 08:01:21.314895 32424 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=8, attempt_no=0}) took 8029 ms. Trace:
I20260812 08:01:21.315047 32424 rpcz_store.cc:276] 0812 08:01:13.284995 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.285048 (+    53us) service_pool.cc:224] Handling call
0812 08:01:21.314872 (+8029824us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:18.883599 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:18.883722 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:18.883727 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:18.883728 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:18.884209 (+   481us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:18.884220 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:18.884224 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:19.908172 (+1023948us) lock_manager.cc:598] Waited 1023650us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:19.908194 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:19.908195 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:19.908243 (+    48us) write_op.cc:270] Start()
0812 08:01:19.908264 (+    21us) write_op.cc:276] Timestamp: P: 1786521679908151 usec, L: 0
0812 08:01:19.908268 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:19.908418 (+   150us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:21.312633 (+1404215us) op_driver.cc:464] REPLICATION: finished
0812 08:01:21.312857 (+   224us) write_op.cc:301] APPLY: starting
0812 08:01:21.312898 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:21.312969 (+    71us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:21.312971 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:21.314284 (+  1313us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:21.314326 (+    42us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:21.314330 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:21.314443 (+   113us) log.cc:844] Serialized 319 byte log entry
0812 08:01:21.314732 (+   289us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:21.314764 (+    32us) write_op.cc:454] Released schema lock
0812 08:01:21.314775 (+    11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":159,"num_ops":50,"prepare.queue_time_us":5598247,"prepare.run_cpu_time_us":1593,"prepare.run_wall_time_us":1024906,"raft.queue_time_us":102,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":507,"replication_time_us":1404324,"row_lock_wait_count":1,"row_lock_wait_us":1023650,"thread_start_us":89,"threads_started":1}]]}
W20260812 08:01:21.316097 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 925ms of latency in SegmentAllocator::Sync()
W20260812 08:01:21.317174 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 958ms of latency in SegmentAllocator::Sync()
W20260812 08:01:21.417454 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:21.507524 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.10]   Dedup: 1.9->[1.10-1.10]
W20260812 08:01:21.509055 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1062ms of latency in SegmentAllocator::Sync()
W20260812 08:01:22.125550 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 11: this message will repeat every 5th retry.
W20260812 08:01:22.275687 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 50) took 959 ms (client timeout 50 ms). Trace:
I20260812 08:01:22.275751 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.275831 31936 rpcz_store.cc:269] 0812 08:01:21.316245 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.316329 (+    84us) service_pool.cc:224] Handling call
0812 08:01:21.316475 (+   146us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:21.316543 (+    68us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:21.316545 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:21.316656 (+   111us) write_op.cc:270] Start()
0812 08:01:21.316676 (+    20us) write_op.cc:276] Timestamp: P: 1786521681314893 usec, L: 0
0812 08:01:21.316953 (+   277us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:21.317014 (+    61us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:21.317020 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.317033 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:22.275560 (+958527us) raft_consensus.cc:1685] finished
0812 08:01:22.275564 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.275657 (+    93us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:21.316980 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:21.317089 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:21.317093 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:21.317095 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:21.317562 (+   467us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:21.317576 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:21.317580 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:21.317852 (+   272us) tablet.cc:701] Row locks acquired
0812 08:01:21.317854 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":221,"prepare.run_cpu_time_us":942,"prepare.run_wall_time_us":941,"thread_start_us":75,"threads_started":1}]]}
I20260812 08:01:22.276111 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.276214 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 51) took 909 ms (client timeout 50 ms). Trace:
I20260812 08:01:22.276333 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.276403 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 52) took 460 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.276448 31939 rpcz_store.cc:269] 0812 08:01:21.815549 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.815759 (+   210us) service_pool.cc:224] Handling call
0812 08:01:22.276077 (+460318us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:22.276212 (+   135us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.276214 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.276222 (+     8us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.276226 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.276236 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.276359 (+   123us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:22.276390 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058196352}
W20260812 08:01:22.276561 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 53) took 410 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.276633 31935 rpcz_store.cc:269] 0812 08:01:21.866437 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.866572 (+   135us) service_pool.cc:224] Handling call
0812 08:01:22.276307 (+409735us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:22.276423 (+   116us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.276427 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.276433 (+     6us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.276437 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.276446 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.276534 (+    88us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:22.276553 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941720448}
W20260812 08:01:22.276371 31938 rpcz_store.cc:269] 0812 08:01:21.367236 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.367314 (+    78us) service_pool.cc:224] Handling call
0812 08:01:22.275701 (+908387us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:22.275950 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.275957 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.275972 (+    15us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.275979 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.275997 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.276143 (+   146us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:22.276193 (+    50us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2088593664}
I20260812 08:01:22.305194 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20260812 08:01:22.308465 32430 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=6, attempt_no=0}) took 9025 ms. Trace:
I20260812 08:01:22.308571 32430 rpcz_store.cc:276] 0812 08:01:13.283393 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.285067 (+  1674us) service_pool.cc:224] Handling call
0812 08:01:22.308425 (+9023358us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:19.908540 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:19.908660 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:19.908665 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:19.908667 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:19.908973 (+   306us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:19.908979 (+     6us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:19.908982 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:21.314908 (+1405926us) lock_manager.cc:598] Waited 1405644us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:21.314930 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:21.314934 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:01:21.314974 (+    40us) write_op.cc:270] Start()
0812 08:01:21.315002 (+    28us) write_op.cc:276] Timestamp: P: 1786521681314893 usec, L: 0
0812 08:01:21.315004 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:21.315158 (+   154us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:22.306177 (+991019us) op_driver.cc:464] REPLICATION: finished
0812 08:01:22.306508 (+   331us) write_op.cc:301] APPLY: starting
0812 08:01:22.306611 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:22.306687 (+    76us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:22.306688 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:22.308042 (+  1354us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:22.308105 (+    63us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:22.308111 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:22.308228 (+   117us) log.cc:844] Serialized 319 byte log entry
0812 08:01:22.308312 (+    84us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:22.308338 (+    26us) write_op.cc:454] Released schema lock
0812 08:01:22.308346 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":29312,"child_traces":[["op",{"apply.queue_time_us":262,"num_ops":50,"prepare.queue_time_us":6623121,"prepare.run_cpu_time_us":1263,"prepare.run_wall_time_us":1406703,"raft.queue_time_us":694,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":165,"replication_time_us":991117,"row_lock_wait_count":1,"row_lock_wait_us":1405644,"spinlock_wait_cycles":1792,"thread_start_us":134,"threads_started":1}]]}
W20260812 08:01:22.308954 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 811ms of latency in SegmentAllocator::Sync()
W20260812 08:01:22.310983 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 902ms of latency in SegmentAllocator::Sync()
W20260812 08:01:22.411545 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:22.571707 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.571779 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 49) took 1064 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.571874 32090 rpcz_store.cc:269] 0812 08:01:21.507105 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.507246 (+   141us) service_pool.cc:224] Handling call
0812 08:01:21.507463 (+   217us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:21.507667 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:21.508106 (+   439us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:21.508309 (+   203us) write_op.cc:270] Start()
0812 08:01:21.508339 (+    30us) write_op.cc:276] Timestamp: P: 1786521681314893 usec, L: 0
0812 08:01:21.508747 (+   408us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:21.508800 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:21.508807 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:21.508823 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:22.571507 (+1062684us) raft_consensus.cc:1685] finished
0812 08:01:22.571510 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.571751 (+   241us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:21.508653 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:21.508793 (+   140us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:21.508797 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:21.508800 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:21.509060 (+   260us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:21.509070 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:21.509072 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:21.510023 (+   951us) tablet.cc:701] Row locks acquired
0812 08:01:21.510025 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":226,"prepare.run_cpu_time_us":1044,"prepare.run_wall_time_us":1437,"row_lock_wait_count":1,"row_lock_wait_us":465,"thread_start_us":116,"threads_started":1}]]}
I20260812 08:01:22.572082 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.572134 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 51) took 547 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.572223 32088 rpcz_store.cc:269] 0812 08:01:22.024488 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.024637 (+   149us) service_pool.cc:224] Handling call
0812 08:01:22.571647 (+547010us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:22.571909 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.571913 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.571925 (+    12us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.571930 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.571943 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.572096 (+   153us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:22.572122 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1257372032}
I20260812 08:01:22.572454 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.10]   Dedup: 1.10->[]
W20260812 08:01:22.572505 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 52) took 497 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.572561 32087 rpcz_store.cc:269] 0812 08:01:22.075410 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.075524 (+   114us) service_pool.cc:224] Handling call
0812 08:01:22.572046 (+496522us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:22.572290 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.572304 (+    14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.572316 (+    12us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.572321 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.572338 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.572460 (+   122us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:22.572491 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141297408}
W20260812 08:01:22.572809 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 50) took 1014 ms (client timeout 50 ms). Trace:
W20260812 08:01:22.572888 32086 rpcz_store.cc:269] 0812 08:01:21.558309 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.558426 (+   117us) service_pool.cc:224] Handling call
0812 08:01:22.572407 (+1013981us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:22.572592 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 9
0812 08:01:22.572598 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:22.572608 (+    10us) raft_consensus.cc:1645] Marking committed up to 9
0812 08:01:22.572612 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.572626 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:22.572766 (+   140us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:22.572796 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2331322624}
I20260812 08:01:22.645946 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.11]   Dedup: 1.10->[1.11-1.11]
W20260812 08:01:22.647346 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1047ms of latency in SegmentAllocator::Sync()
W20260812 08:01:22.695863 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 21 similar messages]
W20260812 08:01:23.213599 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 56) took 903 ms (client timeout 50 ms). Trace:
I20260812 08:01:23.213666 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20260812 08:01:23.214088 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.213717 31938 rpcz_store.cc:269] 0812 08:01:22.310068 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.310139 (+    71us) service_pool.cc:224] Handling call
0812 08:01:22.310270 (+   131us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:22.310334 (+    64us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:22.310336 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:22.310437 (+   101us) write_op.cc:270] Start()
0812 08:01:22.310456 (+    19us) write_op.cc:276] Timestamp: P: 1786521682308492 usec, L: 0
0812 08:01:22.310784 (+   328us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:22.310837 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:22.310842 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.310854 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:23.213466 (+902612us) raft_consensus.cc:1685] finished
0812 08:01:23.213470 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.213571 (+   101us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:22.310788 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:22.310913 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:22.310918 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:22.310920 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:22.311399 (+   479us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:22.311414 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:22.311419 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:22.311843 (+   424us) tablet.cc:701] Row locks acquired
0812 08:01:22.311845 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":249,"prepare.run_cpu_time_us":1119,"prepare.run_wall_time_us":1119,"thread_start_us":136,"threads_started":1}]]}
W20260812 08:01:23.214217 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 59) took 429 ms (client timeout 50 ms). Trace:
W20260812 08:01:23.214270 31936 rpcz_store.cc:269] 0812 08:01:22.784704 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.784798 (+    94us) service_pool.cc:224] Handling call
0812 08:01:23.213613 (+428815us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:23.213965 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.213967 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.213978 (+    11us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.213981 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.213992 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.214164 (+   172us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:23.214203 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":985628032}
W20260812 08:01:23.214421 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 57) took 853 ms (client timeout 50 ms). Trace:
I20260812 08:01:23.214499 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.214517 31935 rpcz_store.cc:269] 0812 08:01:22.361167 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.361288 (+   121us) service_pool.cc:224] Handling call
0812 08:01:23.214060 (+852772us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:23.214258 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.214261 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.214270 (+     9us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.214275 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.214289 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.214382 (+    93us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:23.214407 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1960740096}
W20260812 08:01:23.214816 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 58) took 481 ms (client timeout 50 ms). Trace:
W20260812 08:01:23.214888 31939 rpcz_store.cc:269] 0812 08:01:22.733814 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.733928 (+   114us) service_pool.cc:224] Handling call
0812 08:01:23.214463 (+480535us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:23.214646 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.214649 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.214656 (+     7us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.214660 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.214670 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.214779 (+   109us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:23.214804 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1104517504}
W20260812 08:01:23.309629 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18000
I20260812 08:01:23.357208 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.360792 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1012ms of latency in SegmentAllocator::Sync()
I20260812 08:01:23.361387 32438 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=9, attempt_no=0}) took 10072 ms. Trace:
I20260812 08:01:23.361502 32438 rpcz_store.cc:276] 0812 08:01:13.288455 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.288677 (+   222us) service_pool.cc:224] Handling call
0812 08:01:23.361348 (+10072671us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:21.315266 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:21.315382 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:21.315386 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:21.315389 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:21.315694 (+   305us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:21.315702 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:21.315705 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:22.308477 (+992772us) lock_manager.cc:598] Waited 992489us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:22.308499 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:22.308503 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:01:22.308542 (+    39us) write_op.cc:270] Start()
0812 08:01:22.308565 (+    23us) write_op.cc:276] Timestamp: P: 1786521682308492 usec, L: 0
0812 08:01:22.308567 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:22.308733 (+   166us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:23.358229 (+1049496us) op_driver.cc:464] REPLICATION: finished
0812 08:01:23.358447 (+   218us) write_op.cc:301] APPLY: starting
0812 08:01:23.358484 (+    37us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:23.358571 (+    87us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:23.358572 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:23.360024 (+  1452us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:23.360070 (+    46us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:23.360074 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:23.360177 (+   103us) log.cc:844] Serialized 319 byte log entry
0812 08:01:23.360264 (+    87us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:23.361258 (+   994us) write_op.cc:454] Released schema lock
0812 08:01:23.361267 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":157,"num_ops":50,"prepare.queue_time_us":8025875,"prepare.run_cpu_time_us":1076,"prepare.run_wall_time_us":993549,"raft.queue_time_us":1502,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":187,"replication_time_us":1049623,"row_lock_wait_count":1,"row_lock_wait_us":992489,"thread_start_us":75,"threads_started":1}]]}
W20260812 08:01:23.362206 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1061ms of latency in SegmentAllocator::Sync()
W20260812 08:01:23.462158 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:23.695063 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 53) took 1049 ms (client timeout 50 ms). Trace:
I20260812 08:01:23.695130 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.695199 32086 rpcz_store.cc:269] 0812 08:01:22.645429 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.645559 (+   130us) service_pool.cc:224] Handling call
0812 08:01:22.645875 (+   316us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:22.646252 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:22.646521 (+   269us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:22.646688 (+   167us) write_op.cc:270] Start()
0812 08:01:22.646716 (+    28us) write_op.cc:276] Timestamp: P: 1786521682308492 usec, L: 0
0812 08:01:22.647059 (+   343us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:22.647108 (+    49us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:22.647114 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:22.647129 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:23.694943 (+1047814us) raft_consensus.cc:1685] finished
0812 08:01:23.694946 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.695036 (+    90us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:22.646960 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:22.647094 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:22.647099 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:22.647101 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:22.647346 (+   245us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:22.647355 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:22.647358 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:22.648368 (+  1010us) tablet.cc:701] Row locks acquired
0812 08:01:22.648370 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":159,"prepare.run_cpu_time_us":814,"prepare.run_wall_time_us":1460,"row_lock_wait_count":1,"row_lock_wait_us":722,"thread_start_us":84,"threads_started":1}]]}
I20260812 08:01:23.695437 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.695546 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 54) took 998 ms (client timeout 50 ms). Trace:
W20260812 08:01:23.695592 32087 rpcz_store.cc:269] 0812 08:01:22.697253 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.697387 (+   134us) service_pool.cc:224] Handling call
0812 08:01:23.695079 (+997692us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:23.695306 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.695310 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.695327 (+    17us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.695330 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.695340 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.695492 (+   152us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:23.695530 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2293846784}
W20260812 08:01:23.695739 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 55) took 502 ms (client timeout 50 ms). Trace:
I20260812 08:01:23.695744 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.11]   Dedup: 1.11->[]
W20260812 08:01:23.695799 32088 rpcz_store.cc:269] 0812 08:01:23.193776 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.193894 (+   118us) service_pool.cc:224] Handling call
0812 08:01:23.695406 (+501512us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:23.695563 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.695571 (+     8us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.695578 (+     7us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.695581 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.695592 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.695704 (+   112us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:23.695729 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1152699136}
W20260812 08:01:23.696130 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 56) took 451 ms (client timeout 50 ms). Trace:
I20260812 08:01:23.696278 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.11->[1.12-1.12]
W20260812 08:01:23.696239 32090 rpcz_store.cc:269] 0812 08:01:23.244851 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.244958 (+   107us) service_pool.cc:224] Handling call
0812 08:01:23.695703 (+450745us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:23.695952 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 10
0812 08:01:23.695955 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:23.695963 (+     8us) raft_consensus.cc:1645] Marking committed up to 10
0812 08:01:23.695967 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.695980 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:23.696092 (+   112us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:23.696117 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035923840}
W20260812 08:01:23.697431 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 956ms of latency in SegmentAllocator::Sync()
W20260812 08:01:24.361629 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18840
I20260812 08:01:24.423738 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.423738 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 62) took 1062 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.423920 31939 rpcz_store.cc:269] 0812 08:01:23.361141 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.361226 (+    85us) service_pool.cc:224] Handling call
0812 08:01:23.361365 (+   139us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:23.361437 (+    72us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:23.361440 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:23.361570 (+   130us) write_op.cc:270] Start()
0812 08:01:23.361597 (+    27us) write_op.cc:276] Timestamp: P: 1786521683360289 usec, L: 0
0812 08:01:23.361951 (+   354us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:23.361990 (+    39us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:23.361999 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.362016 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:24.423576 (+1061560us) raft_consensus.cc:1685] finished
0812 08:01:24.423579 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.423687 (+   108us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:23.361866 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:23.362150 (+   284us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:23.362154 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:23.362156 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:23.362478 (+   322us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:23.362487 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:23.362490 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:23.362911 (+   421us) tablet.cc:701] Row locks acquired
0812 08:01:23.362914 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":932,"prepare.run_wall_time_us":1100,"thread_start_us":118,"threads_started":1}]]}
W20260812 08:01:24.424150 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 63) took 1012 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.424230 31935 rpcz_store.cc:269] 0812 08:01:23.412098 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.412198 (+   100us) service_pool.cc:224] Handling call
0812 08:01:24.423686 (+1011488us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.423948 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.423954 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.423966 (+    12us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.423973 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.423989 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.424108 (+   119us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:24.424138 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2325719808}
I20260812 08:01:24.424382 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20260812 08:01:24.424582 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.424664 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 65) took 586 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.424705 31938 rpcz_store.cc:269] 0812 08:01:23.838318 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.838463 (+   145us) service_pool.cc:224] Handling call
0812 08:01:24.424354 (+585891us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.424476 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.424480 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.424490 (+    10us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.424496 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.424511 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.424625 (+   114us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:24.424651 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346931072}
W20260812 08:01:24.424916 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 64) took 638 ms (client timeout 50 ms). Trace:
I20260812 08:01:24.424952 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.424966 31936 rpcz_store.cc:269] 0812 08:01:23.786468 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.786562 (+    94us) service_pool.cc:224] Handling call
0812 08:01:24.424555 (+637993us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.424679 (+   124us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.424681 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.424688 (+     7us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.424768 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.424780 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.424879 (+    99us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:24.424904 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1466732928}
I20260812 08:01:24.425132 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.425246 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 67) took 120 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.425328 31942 rpcz_store.cc:269] 0812 08:01:24.305037 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.305176 (+   139us) service_pool.cc:224] Handling call
0812 08:01:24.424930 (+119754us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.425028 (+    98us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.425031 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.425037 (+     6us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.425040 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.425049 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.425157 (+   108us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:24.425183 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274978176}
W20260812 08:01:24.425400 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 66) took 171 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.425632 31941 rpcz_store.cc:269] 0812 08:01:24.253981 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.254081 (+   100us) service_pool.cc:224] Handling call
0812 08:01:24.425098 (+171017us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.425239 (+   141us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.425242 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.425250 (+     8us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.425253 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.425263 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.425367 (+   104us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:24.425391 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":392796032}
W20260812 08:01:24.585520 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 16: this message will repeat every 5th retry.
I20260812 08:01:24.654150 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.654243 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 58) took 981 ms (client timeout 50 ms). Trace:
I20260812 08:01:24.654728 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.654788 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 60) took 532 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.654425 32089 rpcz_store.cc:269] 0812 08:01:23.673179 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.673310 (+   131us) service_pool.cc:224] Handling call
0812 08:01:23.696150 (+ 22840us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:23.696420 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:23.696658 (+   238us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:23.696805 (+   147us) write_op.cc:270] Start()
0812 08:01:23.696836 (+    31us) write_op.cc:276] Timestamp: P: 1786521683360289 usec, L: 0
0812 08:01:23.697163 (+   327us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:23.697205 (+    42us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:23.697211 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:23.697223 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:24.653889 (+956666us) raft_consensus.cc:1685] finished
0812 08:01:24.653892 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.654133 (+   241us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.654217 (+    84us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:23.697075 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:23.697206 (+   131us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:23.697210 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:23.697212 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:23.697486 (+   274us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:23.697522 (+    36us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:23.697528 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:23.698495 (+   967us) tablet.cc:701] Row locks acquired
0812 08:01:23.698497 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":51812224,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":27,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":1473,"row_lock_wait_count":1,"row_lock_wait_us":714,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:01:24.654848 32088 rpcz_store.cc:269] 0812 08:01:24.122376 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.122464 (+    88us) service_pool.cc:224] Handling call
0812 08:01:24.654008 (+531544us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:24.654472 (+   464us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.654476 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.654488 (+    12us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.654503 (+    15us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.654517 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.654724 (+   207us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.654773 (+    49us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1221762304}
I20260812 08:01:24.655016 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.655098 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 59) took 583 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.655196 32090 rpcz_store.cc:269] 0812 08:01:24.071167 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.071273 (+   106us) service_pool.cc:224] Handling call
0812 08:01:24.654695 (+583422us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:24.654876 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.654882 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.654893 (+    11us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.654900 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.654913 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.655057 (+   144us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.655086 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1341160192}
W20260812 08:01:24.655405 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 57) took 1033 ms (client timeout 50 ms). Trace:
I20260812 08:01:24.655539 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.655478 32085 rpcz_store.cc:269] 0812 08:01:23.622174 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.622468 (+   294us) service_pool.cc:224] Handling call
0812 08:01:24.654987 (+1032519us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:24.655226 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.655229 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.655236 (+     7us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.655240 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.655270 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.655370 (+   100us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.655395 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2373790464}
I20260812 08:01:24.655784 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.655843 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 61) took 171 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.655879 32087 rpcz_store.cc:269] 0812 08:01:24.484334 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.484480 (+   146us) service_pool.cc:224] Handling call
0812 08:01:24.655509 (+171029us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:24.655650 (+   141us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.655653 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.655689 (+    36us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.655696 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.655709 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.655810 (+   101us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.655831 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":392616192}
W20260812 08:01:24.656081 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 62) took 120 ms (client timeout 50 ms). Trace:
W20260812 08:01:24.656134 32086 rpcz_store.cc:269] 0812 08:01:24.535379 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.535663 (+   284us) service_pool.cc:224] Handling call
0812 08:01:24.655753 (+120090us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:24.655893 (+   140us) raft_consensus.cc:1534] Early marking committed up to index 11
0812 08:01:24.655899 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:24.655906 (+     7us) raft_consensus.cc:1645] Marking committed up to 11
0812 08:01:24.655909 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.655918 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:24.656029 (+   111us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:24.656059 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275622144}
I20260812 08:01:24.816185 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
W20260812 08:01:24.820432 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 967ms of latency in SegmentAllocator::Sync()
I20260812 08:01:24.821810 32446 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=10, attempt_no=0}) took 11533 ms. Trace:
I20260812 08:01:24.821952 32446 rpcz_store.cc:276] 0812 08:01:13.288580 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.288958 (+   378us) service_pool.cc:224] Handling call
0812 08:01:24.821791 (+11532833us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:22.308849 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:22.308975 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:22.308981 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:22.308984 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:22.309252 (+   268us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:22.309260 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:22.309263 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:23.360298 (+1051035us) lock_manager.cc:598] Waited 1050784us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:23.360310 (+    12us) tablet.cc:701] Row locks acquired
0812 08:01:23.360312 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:23.360359 (+    47us) write_op.cc:270] Start()
0812 08:01:23.360379 (+    20us) write_op.cc:276] Timestamp: P: 1786521683360289 usec, L: 0
0812 08:01:23.360381 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:23.360535 (+   154us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:24.817691 (+1457156us) op_driver.cc:464] REPLICATION: finished
0812 08:01:24.818004 (+   313us) write_op.cc:301] APPLY: starting
0812 08:01:24.818039 (+    35us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:24.818113 (+    74us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:24.818114 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:24.819676 (+  1562us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:24.819731 (+    55us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:24.819738 (+     7us) write_op.cc:312] APPLY: finished
0812 08:01:24.819886 (+   148us) log.cc:844] Serialized 319 byte log entry
0812 08:01:24.819967 (+    81us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:24.821702 (+  1735us) write_op.cc:454] Released schema lock
0812 08:01:24.821710 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"num_ops":50,"prepare.queue_time_us":9019303,"prepare.run_cpu_time_us":1219,"prepare.run_wall_time_us":1051778,"raft.queue_time_us":267,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":463,"replication_time_us":1457271,"row_lock_wait_count":1,"row_lock_wait_us":1050784,"thread_start_us":80,"threads_started":1}]]}
W20260812 08:01:24.823961 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 955ms of latency in SegmentAllocator::Sync()
W20260812 08:01:24.923309 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:24.968362 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.13]   Dedup: 1.12->[1.13-1.13]
W20260812 08:01:24.969583 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1171ms of latency in SegmentAllocator::Sync()
W20260812 08:01:25.779600 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 70) took 958 ms (client timeout 50 ms). Trace:
I20260812 08:01:25.779639 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20260812 08:01:25.779840 31941 rpcz_store.cc:269] 0812 08:01:24.821601 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.822042 (+   441us) service_pool.cc:224] Handling call
0812 08:01:24.822206 (+   164us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:24.822310 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:24.822315 (+     5us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:24.822845 (+   530us) write_op.cc:270] Start()
0812 08:01:24.822901 (+    56us) write_op.cc:276] Timestamp: P: 1786521684820002 usec, L: 0
0812 08:01:24.823494 (+   593us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:24.823566 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:24.823573 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.823586 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:25.779459 (+955873us) raft_consensus.cc:1685] finished
0812 08:01:25.779462 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.779567 (+   105us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:24.823303 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:24.823423 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:24.823427 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:24.823429 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:24.824519 (+  1090us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:24.824533 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:24.824537 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:24.824802 (+   265us) tablet.cc:701] Row locks acquired
0812 08:01:24.824804 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":295,"prepare.run_cpu_time_us":1561,"prepare.run_wall_time_us":1560,"thread_start_us":342,"threads_started":1}]]}
I20260812 08:01:25.780334 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20260812 08:01:25.780383 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 72) took 521 ms (client timeout 50 ms). Trace:
W20260812 08:01:25.780444 31936 rpcz_store.cc:269] 0812 08:01:25.258539 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.258722 (+   183us) service_pool.cc:224] Handling call
0812 08:01:25.779588 (+520866us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.780105 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:25.780108 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:25.780120 (+    12us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:25.780123 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.780139 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.780327 (+   188us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:25.780363 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1197391744}
I20260812 08:01:25.780606 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20260812 08:01:25.780702 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 71) took 907 ms (client timeout 50 ms). Trace:
W20260812 08:01:25.780758 31942 rpcz_store.cc:269] 0812 08:01:24.873078 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.873237 (+   159us) service_pool.cc:224] Handling call
0812 08:01:25.780295 (+907058us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.780469 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:25.780475 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:25.780487 (+    12us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:25.780494 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.780506 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.780656 (+   150us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:25.780688 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2085584256}
I20260812 08:01:25.780860 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20260812 08:01:25.780894 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 74) took 98 ms (client timeout 50 ms). Trace:
W20260812 08:01:25.780951 31935 rpcz_store.cc:269] 0812 08:01:25.682344 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.682476 (+   132us) service_pool.cc:224] Handling call
0812 08:01:25.780577 (+ 98101us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.780724 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:25.780728 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:25.780735 (+     7us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:25.780739 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.780753 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.780862 (+   109us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:25.780885 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224964352}
I20260812 08:01:25.781057 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13]   Dedup: 1.13->[]
W20260812 08:01:25.781136 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 73) took 471 ms (client timeout 50 ms). Trace:
W20260812 08:01:25.781190 31938 rpcz_store.cc:269] 0812 08:01:25.309604 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.309717 (+   113us) service_pool.cc:224] Handling call
0812 08:01:25.780832 (+471115us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.780958 (+   126us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:25.780963 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:25.780971 (+     8us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:25.780977 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.780990 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.781097 (+   107us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:25.781126 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1083002752}
W20260812 08:01:25.781405 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 75) took 48 ms (client timeout 50 ms). Trace:
W20260812 08:01:25.781504 31939 rpcz_store.cc:269] 0812 08:01:25.733313 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.733406 (+    93us) service_pool.cc:224] Handling call
0812 08:01:25.781032 (+ 47626us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.781142 (+   110us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:25.781148 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:25.781156 (+     8us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:25.781159 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.781243 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:25.781361 (+   118us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:25.781389 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109096832}
I20260812 08:01:25.790853 32453 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=11, attempt_no=0}) took 12501 ms. Trace:
I20260812 08:01:25.791059 32453 rpcz_store.cc:276] 0812 08:01:13.289816 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.289864 (+    48us) service_pool.cc:224] Handling call
0812 08:01:25.790812 (+12500948us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:23.360666 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:23.360801 (+   135us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:23.360807 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:23.360810 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:23.361206 (+   396us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:23.361215 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:23.361218 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:24.820003 (+1458785us) lock_manager.cc:598] Waited 1458506us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:24.820028 (+    25us) tablet.cc:701] Row locks acquired
0812 08:01:24.820032 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:01:24.820079 (+    47us) write_op.cc:270] Start()
0812 08:01:24.820104 (+    25us) write_op.cc:276] Timestamp: P: 1786521684820002 usec, L: 0
0812 08:01:24.820106 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:24.820267 (+   161us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:25.788012 (+967745us) op_driver.cc:464] REPLICATION: finished
0812 08:01:25.788371 (+   359us) write_op.cc:301] APPLY: starting
0812 08:01:25.788417 (+    46us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:25.788528 (+   111us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:25.788529 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:25.790152 (+  1623us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:25.790205 (+    53us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:25.790209 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:25.790374 (+   165us) log.cc:844] Serialized 319 byte log entry
0812 08:01:25.790649 (+   275us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:25.790712 (+    63us) write_op.cc:454] Released schema lock
0812 08:01:25.790722 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":267,"num_ops":50,"prepare.queue_time_us":10070528,"prepare.run_cpu_time_us":1378,"prepare.run_wall_time_us":1459680,"raft.queue_time_us":528,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":5,"replication_time_us":967866,"row_lock_wait_count":1,"row_lock_wait_us":1458506,"spinlock_wait_cycles":19200,"thread_start_us":124,"threads_started":1}]]}
W20260812 08:01:25.792133 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1152ms of latency in SegmentAllocator::Sync()
W20260812 08:01:25.793331 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 991ms of latency in SegmentAllocator::Sync()
W20260812 08:01:25.893522 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:26.141237 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.13]   Dedup: 1.13->[]
W20260812 08:01:26.141278 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 63) took 1173 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.141462 32086 rpcz_store.cc:269] 0812 08:01:24.967807 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.967996 (+   189us) service_pool.cc:224] Handling call
0812 08:01:24.968248 (+   252us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:24.968577 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:24.968782 (+   205us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:24.968910 (+   128us) write_op.cc:270] Start()
0812 08:01:24.968938 (+    28us) write_op.cc:276] Timestamp: P: 1786521684820002 usec, L: 0
0812 08:01:24.969302 (+   364us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:24.969348 (+    46us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:24.969354 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:24.969369 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:26.141061 (+1171692us) raft_consensus.cc:1685] finished
0812 08:01:26.141064 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.141242 (+   178us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:24.969200 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:24.969333 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:24.969338 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:24.969339 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:24.969587 (+   248us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:24.969597 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:24.969599 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:24.970779 (+  1180us) tablet.cc:701] Row locks acquired
0812 08:01:24.970782 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":177,"prepare.run_cpu_time_us":895,"prepare.run_wall_time_us":1641,"row_lock_wait_count":1,"row_lock_wait_us":881,"thread_start_us":103,"threads_started":1}]]}
W20260812 08:01:26.141788 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 65) took 589 ms (client timeout 50 ms). Trace:
I20260812 08:01:26.141726 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.13]   Dedup: 1.13->[]
W20260812 08:01:26.141841 32085 rpcz_store.cc:269] 0812 08:01:25.552574 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.552701 (+   127us) service_pool.cc:224] Handling call
0812 08:01:26.141178 (+588477us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:26.141493 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:26.141497 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.141510 (+    13us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:26.141514 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.141528 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.141671 (+   143us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:26.141772 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1352775552}
I20260812 08:01:26.142024 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.13]   Dedup: 1.13->[]
W20260812 08:01:26.142367 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 64) took 1122 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.142436 32087 rpcz_store.cc:269] 0812 08:01:25.019579 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.019705 (+   126us) service_pool.cc:224] Handling call
0812 08:01:26.141685 (+1121980us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:26.141897 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:26.141904 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.141916 (+    12us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:26.141924 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.141941 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.142070 (+   129us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:26.142323 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2579633920}
I20260812 08:01:26.142656 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.13->[1.14-1.14]
W20260812 08:01:26.142730 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 66) took 539 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.142802 32090 rpcz_store.cc:269] 0812 08:01:25.603679 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.603790 (+   111us) service_pool.cc:224] Handling call
0812 08:01:26.141990 (+538200us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:26.142368 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 12
0812 08:01:26.142371 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.142381 (+    10us) raft_consensus.cc:1645] Marking committed up to 12
0812 08:01:26.142388 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.142401 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.142679 (+   278us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:26.142713 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1237048704}
W20260812 08:01:26.143816 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 952ms of latency in SegmentAllocator::Sync()
W20260812 08:01:26.784982 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 77) took 992 ms (client timeout 50 ms). Trace:
I20260812 08:01:26.785004 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
W20260812 08:01:26.785249 31939 rpcz_store.cc:269] 0812 08:01:25.792339 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.792409 (+    70us) service_pool.cc:224] Handling call
0812 08:01:25.792567 (+   158us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:25.792635 (+    68us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:25.792637 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:25.792783 (+   146us) write_op.cc:270] Start()
0812 08:01:25.792807 (+    24us) write_op.cc:276] Timestamp: P: 1786521685790698 usec, L: 0
0812 08:01:25.793082 (+   275us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:25.793158 (+    76us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:25.793165 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:25.793178 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:26.784841 (+991663us) raft_consensus.cc:1685] finished
0812 08:01:26.784844 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.784954 (+   110us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:25.793077 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:25.793194 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:25.793198 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:25.793200 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:25.793519 (+   319us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:25.793530 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:25.793533 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:25.793784 (+   251us) tablet.cc:701] Row locks acquired
0812 08:01:25.793785 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":761,"thread_start_us":90,"threads_started":1}]]}
I20260812 08:01:26.785522 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
W20260812 08:01:26.785597 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 78) took 942 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.785642 31938 rpcz_store.cc:269] 0812 08:01:25.843286 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.843387 (+   101us) service_pool.cc:224] Handling call
0812 08:01:26.784959 (+941572us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:26.785390 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:26.785393 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.785403 (+    10us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:26.785406 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.785417 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.785543 (+   126us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:26.785582 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2164954880}
I20260812 08:01:26.785795 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
W20260812 08:01:26.785869 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 79) took 521 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.786033 31935 rpcz_store.cc:269] 0812 08:01:26.264613 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.264721 (+   108us) service_pool.cc:224] Handling call
0812 08:01:26.785491 (+520770us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:26.785620 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:26.785623 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.785629 (+     6us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:26.785633 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.785643 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.785751 (+   108us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:26.785853 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1197161216}
W20260812 08:01:26.786212 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 80) took 470 ms (client timeout 50 ms). Trace:
W20260812 08:01:26.786295 31942 rpcz_store.cc:269] 0812 08:01:26.315794 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.315903 (+   109us) service_pool.cc:224] Handling call
0812 08:01:26.785757 (+469854us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:26.786011 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:26.786014 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:26.786022 (+     8us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:26.786026 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.786037 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:26.786153 (+   116us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:26.786197 (+    44us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080117632}
W20260812 08:01:26.792107 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18dc0
I20260812 08:01:26.831559 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14]   Dedup: 1.14->[]
I20260812 08:01:26.947294 32460 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=12, attempt_no=0}) took 13654 ms. Trace:
W20260812 08:01:26.947576 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 951ms of latency in SegmentAllocator::Sync()
I20260812 08:01:26.947407 32460 rpcz_store.cc:276] 0812 08:01:13.292482 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.292533 (+    51us) service_pool.cc:224] Handling call
0812 08:01:26.947262 (+13654729us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:24.820386 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:24.820527 (+   141us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:24.820531 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:24.820533 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:24.821541 (+  1008us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:24.821551 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:24.821554 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:25.790716 (+969162us) lock_manager.cc:598] Waited 968756us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:25.790729 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:25.790731 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:25.790774 (+    43us) write_op.cc:270] Start()
0812 08:01:25.790796 (+    22us) write_op.cc:276] Timestamp: P: 1786521685790698 usec, L: 0
0812 08:01:25.790798 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:25.790946 (+   148us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:26.944812 (+1153866us) op_driver.cc:464] REPLICATION: finished
0812 08:01:26.945166 (+   354us) write_op.cc:301] APPLY: starting
0812 08:01:26.945203 (+    37us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:26.945265 (+    62us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:26.945266 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:26.946666 (+  1400us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:26.946733 (+    67us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:26.946739 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:26.946879 (+   140us) log.cc:844] Serialized 319 byte log entry
0812 08:01:26.947118 (+   239us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:26.947169 (+    51us) write_op.cc:454] Released schema lock
0812 08:01:26.947177 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":287,"num_ops":50,"prepare.queue_time_us":11527531,"prepare.run_cpu_time_us":1968,"prepare.run_wall_time_us":970658,"raft.queue_time_us":1508,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"replication_time_us":1153959,"row_lock_wait_count":1,"row_lock_wait_us":968756,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}]]}
W20260812 08:01:26.949066 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1125ms of latency in SegmentAllocator::Sync()
W20260812 08:01:27.037365 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 21: this message will repeat every 5th retry.
W20260812 08:01:27.049669 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:27.096457 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.14->[]
W20260812 08:01:27.096526 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 67) took 1113 ms (client timeout 50 ms). Trace:
W20260812 08:01:27.096843 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 69) took 671 ms (client timeout 50 ms). Trace:
I20260812 08:01:27.096865 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.14->[]
W20260812 08:01:27.096659 32088 rpcz_store.cc:269] 0812 08:01:25.983252 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.983416 (+   164us) service_pool.cc:224] Handling call
0812 08:01:26.142517 (+159101us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:26.142822 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:26.143032 (+   210us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:26.143140 (+   108us) write_op.cc:270] Start()
0812 08:01:26.143182 (+    42us) write_op.cc:276] Timestamp: P: 1786521685790698 usec, L: 0
0812 08:01:26.143505 (+   323us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:26.143543 (+    38us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:26.143549 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.143560 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:27.096264 (+952704us) raft_consensus.cc:1685] finished
0812 08:01:27.096267 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.096446 (+   179us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.096502 (+    56us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:26.143474 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:26.143633 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:26.143638 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:26.143639 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:26.144034 (+   395us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:26.144045 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:26.144048 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:26.144668 (+   620us) tablet.cc:701] Row locks acquired
0812 08:01:26.144670 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":365213056,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":193,"prepare.run_cpu_time_us":889,"prepare.run_wall_time_us":1251,"row_lock_wait_count":1,"row_lock_wait_us":421,"thread_start_us":86,"threads_started":1}]]}
W20260812 08:01:27.096927 32090 rpcz_store.cc:269] 0812 08:01:26.425729 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.425810 (+    81us) service_pool.cc:224] Handling call
0812 08:01:27.096397 (+670587us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:27.096637 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:27.096639 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:27.096649 (+    10us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:27.096676 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.096695 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.096799 (+   104us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.096830 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1541512448}
W20260812 08:01:27.097169 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 70) took 620 ms (client timeout 50 ms). Trace:
I20260812 08:01:27.097249 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.14->[]
W20260812 08:01:27.097235 32087 rpcz_store.cc:269] 0812 08:01:26.476692 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.476802 (+   110us) service_pool.cc:224] Handling call
0812 08:01:27.096824 (+620022us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:27.096975 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:27.096981 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:27.096991 (+    10us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:27.096995 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.097006 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.097125 (+   119us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.097154 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1425285376}
I20260812 08:01:27.097477 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.14->[]
W20260812 08:01:27.097558 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 68) took 1063 ms (client timeout 50 ms). Trace:
W20260812 08:01:27.097640 32089 rpcz_store.cc:269] 0812 08:01:26.034368 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.034489 (+   121us) service_pool.cc:224] Handling call
0812 08:01:27.097210 (+1062721us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:27.097359 (+   149us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:27.097363 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:27.097371 (+     8us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:27.097375 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.097389 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.097512 (+   123us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.097543 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443179776}
I20260812 08:01:27.097719 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.14]   Dedup: 1.14->[]
W20260812 08:01:27.097785 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 72) took 110 ms (client timeout 50 ms). Trace:
W20260812 08:01:27.097868 32086 rpcz_store.cc:269] 0812 08:01:26.987370 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.987484 (+   114us) service_pool.cc:224] Handling call
0812 08:01:27.097446 (+109962us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:27.097594 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:27.097597 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:27.097605 (+     8us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:27.097609 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.097618 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.097739 (+   121us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.097770 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252174080}
W20260812 08:01:27.098001 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 71) took 161 ms (client timeout 50 ms). Trace:
W20260812 08:01:27.098063 32085 rpcz_store.cc:269] 0812 08:01:26.936259 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.936341 (+    82us) service_pool.cc:224] Handling call
0812 08:01:27.097691 (+161350us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:27.097813 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 13
0812 08:01:27.097817 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:27.097824 (+     7us) raft_consensus.cc:1645] Marking committed up to 13
0812 08:01:27.097828 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.097843 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:27.097959 (+   116us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:27.097987 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370462464}
I20260812 08:01:27.423079 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.14->[1.15-1.15]
W20260812 08:01:27.424350 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1116ms of latency in SegmentAllocator::Sync()
W20260812 08:01:27.927503 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 21 similar messages]
W20260812 08:01:27.948657 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9340
W20260812 08:01:28.074748 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 83) took 1126 ms (client timeout 50 ms). Trace:
I20260812 08:01:28.074796 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.074934 31942 rpcz_store.cc:269] 0812 08:01:26.947997 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.948259 (+   262us) service_pool.cc:224] Handling call
0812 08:01:26.948403 (+   144us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:26.948479 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:26.948482 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:26.948562 (+    80us) write_op.cc:270] Start()
0812 08:01:26.948588 (+    26us) write_op.cc:276] Timestamp: P: 1786521686947163 usec, L: 0
0812 08:01:26.948896 (+   308us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:26.948955 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:26.948961 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:26.948974 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:28.074550 (+1125576us) raft_consensus.cc:1685] finished
0812 08:01:28.074552 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.074716 (+   164us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:26.950733 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:26.950868 (+   135us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:26.950872 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:26.950873 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:26.951155 (+   282us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:26.951166 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:26.951169 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:26.951358 (+   189us) tablet.cc:701] Row locks acquired
0812 08:01:26.951359 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":199,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":2526,"thread_start_us":67,"threads_started":1}]]}
I20260812 08:01:28.075310 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.075377 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 84) took 1076 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.075423 31935 rpcz_store.cc:269] 0812 08:01:26.999429 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.999540 (+   111us) service_pool.cc:224] Handling call
0812 08:01:28.074748 (+1075208us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.075146 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.075152 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.075167 (+    15us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.075175 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.075189 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.075323 (+   134us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:28.075363 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2471708928}
I20260812 08:01:28.075556 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.075603 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 87) took 113 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.075645 31936 rpcz_store.cc:269] 0812 08:01:27.961775 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.961912 (+   137us) service_pool.cc:224] Handling call
0812 08:01:28.075279 (+113367us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.075414 (+   135us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.075417 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.075426 (+     9us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.075433 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.075443 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.075568 (+   125us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:28.075590 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":260253952}
I20260812 08:01:28.075752 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
I20260812 08:01:28.075956 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.076077 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 85) took 720 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.076181 31938 rpcz_store.cc:269] 0812 08:01:27.355283 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.355381 (+    98us) service_pool.cc:224] Handling call
0812 08:01:28.075733 (+720352us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.075847 (+   114us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.075851 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.075860 (+     9us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.075867 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.075881 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.076012 (+   131us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:28.076061 (+    49us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1656267520}
W20260812 08:01:28.076272 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 88) took 63 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.076458 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 86) took 670 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.076378 31941 rpcz_store.cc:269] 0812 08:01:28.012868 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.013008 (+   140us) service_pool.cc:224] Handling call
0812 08:01:28.075931 (+ 62923us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.076066 (+   135us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.076068 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.076074 (+     6us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.076076 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.076084 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.076229 (+   145us) spinlock_profiling.cc:243] Waited 62.7 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:28.076254 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144242048}
W20260812 08:01:28.076546 31939 rpcz_store.cc:269] 0812 08:01:27.406240 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.406344 (+   104us) service_pool.cc:224] Handling call
0812 08:01:28.075524 (+669180us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.075670 (+   146us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.075675 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.075684 (+     9us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.075688 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.075698 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.076430 (+   732us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:28.076449 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1538449152}
I20260812 08:01:28.540958 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.540959 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 73) took 1118 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.541209 32085 rpcz_store.cc:269] 0812 08:01:27.422416 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.422519 (+   103us) service_pool.cc:224] Handling call
0812 08:01:27.422988 (+   469us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:27.423270 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:27.423520 (+   250us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:27.423631 (+   111us) write_op.cc:270] Start()
0812 08:01:27.423665 (+    34us) write_op.cc:276] Timestamp: P: 1786521686947163 usec, L: 0
0812 08:01:27.424002 (+   337us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:27.424062 (+    60us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:27.424069 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:27.424085 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:28.540779 (+1116694us) raft_consensus.cc:1685] finished
0812 08:01:28.540782 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.540923 (+   141us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:27.424183 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:27.424314 (+   131us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:27.424318 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:27.424321 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:27.424741 (+   420us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:27.424755 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:27.424759 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:27.425469 (+   710us) tablet.cc:701] Row locks acquired
0812 08:01:27.425471 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":426,"prepare.run_cpu_time_us":1135,"prepare.run_wall_time_us":1346,"row_lock_wait_count":1,"row_lock_wait_us":261,"thread_start_us":72,"threads_started":1}]]}
I20260812 08:01:28.541460 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.541476 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 76) took 612 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.541607 32087 rpcz_store.cc:269] 0812 08:01:27.928646 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.928789 (+   143us) service_pool.cc:224] Handling call
0812 08:01:28.540906 (+612117us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:28.541253 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.541257 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.541269 (+    12us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.541273 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.541287 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.541431 (+   144us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:28.541462 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1406919040}
I20260812 08:01:28.541772 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.541921 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 74) took 1068 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.541972 32086 rpcz_store.cc:269] 0812 08:01:27.473368 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.473495 (+   127us) service_pool.cc:224] Handling call
0812 08:01:28.541422 (+1067927us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:28.541633 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.541640 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.541652 (+    12us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.541659 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.541674 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.541867 (+   193us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:28.541906 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2455297280}
I20260812 08:01:28.542050 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.542098 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 77) took 207 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.542142 32090 rpcz_store.cc:269] 0812 08:01:28.334894 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.335015 (+   121us) service_pool.cc:224] Handling call
0812 08:01:28.541743 (+206728us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:28.541935 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.541941 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.541949 (+     8us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.541953 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.541962 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.542068 (+   106us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:28.542089 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474669312}
I20260812 08:01:28.542277 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.15]   Dedup: 1.15->[]
W20260812 08:01:28.542348 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 78) took 156 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.542393 32088 rpcz_store.cc:269] 0812 08:01:28.386131 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.386235 (+   104us) service_pool.cc:224] Handling call
0812 08:01:28.542021 (+155786us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:28.542157 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.542161 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.542168 (+     7us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.542172 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.542182 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.542299 (+   117us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:28.542335 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":357516544}
W20260812 08:01:28.542520 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 75) took 665 ms (client timeout 50 ms). Trace:
W20260812 08:01:28.542564 32089 rpcz_store.cc:269] 0812 08:01:27.877467 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:27.877658 (+   191us) service_pool.cc:224] Handling call
0812 08:01:28.542241 (+664583us) raft_consensus.cc:1471] Updating replica for 10 ops
0812 08:01:28.542373 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 14
0812 08:01:28.542375 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:28.542382 (+     7us) raft_consensus.cc:1645] Marking committed up to 14
0812 08:01:28.542386 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.542397 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:28.542492 (+    95us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:28.542510 (+    18us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527504768}
I20260812 08:01:28.556437 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15]   Dedup: 1.15->[]
I20260812 08:01:28.559823 32469 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=15, attempt_no=0}) took 15266 ms. Trace:
I20260812 08:01:28.559979 32469 rpcz_store.cc:276] 0812 08:01:13.293371 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.293412 (+    41us) service_pool.cc:224] Handling call
0812 08:01:28.559801 (+15266389us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:25.791078 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:25.791193 (+   115us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:25.791209 (+    16us) write_op.cc:435] Acquired schema lock
0812 08:01:25.791211 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:25.791639 (+   428us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:25.791647 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:25.791650 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:26.947161 (+1155511us) lock_manager.cc:598] Waited 1155185us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:26.947173 (+    12us) tablet.cc:701] Row locks acquired
0812 08:01:26.947175 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:26.947231 (+    56us) write_op.cc:270] Start()
0812 08:01:26.947253 (+    22us) write_op.cc:276] Timestamp: P: 1786521686947163 usec, L: 0
0812 08:01:26.947255 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:26.947407 (+   152us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:28.557604 (+1610197us) op_driver.cc:464] REPLICATION: finished
0812 08:01:28.557896 (+   292us) write_op.cc:301] APPLY: starting
0812 08:01:28.557938 (+    42us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:28.558016 (+    78us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:28.558018 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:28.559433 (+  1415us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:28.559474 (+    41us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:28.559478 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:28.559607 (+   129us) log.cc:844] Serialized 319 byte log entry
0812 08:01:28.559677 (+    70us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:28.559705 (+    28us) write_op.cc:454] Released schema lock
0812 08:01:28.559714 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226,"num_ops":50,"prepare.queue_time_us":12497414,"prepare.run_cpu_time_us":1417,"prepare.run_wall_time_us":1156416,"raft.queue_time_us":60,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"replication_time_us":1610306,"row_lock_wait_count":1,"row_lock_wait_us":1155185,"thread_start_us":88,"threads_started":1}]]}
W20260812 08:01:28.560293 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1075ms of latency in SegmentAllocator::Sync()
W20260812 08:01:28.562888 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1021ms of latency in SegmentAllocator::Sync()
W20260812 08:01:28.662721 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:28.914570 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.16]   Dedup: 1.15->[1.16-1.16]
W20260812 08:01:28.915921 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1074ms of latency in SegmentAllocator::Sync()
W20260812 08:01:29.400593 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 26: this message will repeat every 5th retry.
W20260812 08:01:29.561146 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d182c0
W20260812 08:01:29.584481 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 91) took 1023 ms (client timeout 50 ms). Trace:
I20260812 08:01:29.584576 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.584642 31939 rpcz_store.cc:269] 0812 08:01:28.561435 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.561504 (+    69us) service_pool.cc:224] Handling call
0812 08:01:28.561635 (+   131us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:28.561707 (+    72us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:28.561709 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:28.561815 (+   106us) write_op.cc:270] Start()
0812 08:01:28.561836 (+    21us) write_op.cc:276] Timestamp: P: 1786521688559851 usec, L: 0
0812 08:01:28.562094 (+   258us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:28.562121 (+    27us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:28.562125 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.562135 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:29.584346 (+1022211us) raft_consensus.cc:1685] finished
0812 08:01:29.584350 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.584455 (+   105us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:28.562100 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:28.562207 (+   107us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:28.562211 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:28.562214 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:28.562561 (+   347us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:28.562572 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:28.562607 (+    35us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:28.562863 (+   256us) tablet.cc:701] Row locks acquired
0812 08:01:28.562865 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":858,"prepare.run_wall_time_us":858,"thread_start_us":87,"threads_started":1}]]}
I20260812 08:01:29.585095 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.585155 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 92) took 972 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.585302 31941 rpcz_store.cc:269] 0812 08:01:28.612617 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.612714 (+    97us) service_pool.cc:224] Handling call
0812 08:01:29.584473 (+971759us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:29.584895 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.584902 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.584918 (+    16us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.584926 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.584944 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.585098 (+   154us) spinlock_profiling.cc:243] Waited 971 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:29.585139 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2234336640}
W20260812 08:01:29.585479 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 93) took 459 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.585556 31938 rpcz_store.cc:269] 0812 08:01:29.125795 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.126006 (+   211us) service_pool.cc:224] Handling call
0812 08:01:29.585060 (+459054us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:29.585262 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.585266 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.585274 (+     8us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.585277 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.585291 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.585440 (+   149us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:29.585467 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055107328}
I20260812 08:01:29.585613 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.585927 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 94) took 409 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.585978 31936 rpcz_store.cc:269] 0812 08:01:29.176722 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.176854 (+   132us) service_pool.cc:224] Handling call
0812 08:01:29.585588 (+408734us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:29.585753 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.585760 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.585771 (+    11us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.585779 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.585792 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.585895 (+   103us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:29.585916 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939364096}
I20260812 08:01:29.659888 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.664119 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1091ms of latency in SegmentAllocator::Sync()
I20260812 08:01:29.664521 32479 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=14, attempt_no=0}) took 16370 ms. Trace:
I20260812 08:01:29.664633 32479 rpcz_store.cc:276] 0812 08:01:13.293306 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.293682 (+   376us) service_pool.cc:224] Handling call
0812 08:01:29.664191 (+16370509us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:26.947522 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:26.947644 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:26.947650 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:26.947653 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:26.948173 (+   520us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:26.948186 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:26.948192 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:28.559847 (+1611655us) lock_manager.cc:598] Waited 1611322us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:28.559861 (+    14us) tablet.cc:701] Row locks acquired
0812 08:01:28.559864 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:01:28.559908 (+    44us) write_op.cc:270] Start()
0812 08:01:28.559933 (+    25us) write_op.cc:276] Timestamp: P: 1786521688559851 usec, L: 0
0812 08:01:28.559935 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:28.560087 (+   152us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:29.661431 (+1101344us) op_driver.cc:464] REPLICATION: finished
0812 08:01:29.661669 (+   238us) write_op.cc:301] APPLY: starting
0812 08:01:29.661785 (+   116us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:29.661866 (+    81us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:29.661871 (+     5us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:29.663273 (+  1402us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:29.663338 (+    65us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:29.663346 (+     8us) write_op.cc:312] APPLY: finished
0812 08:01:29.663470 (+   124us) log.cc:844] Serialized 319 byte log entry
0812 08:01:29.663540 (+    70us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:29.664110 (+   570us) write_op.cc:454] Released schema lock
0812 08:01:29.664120 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":171,"num_ops":50,"prepare.queue_time_us":13653604,"prepare.run_cpu_time_us":1563,"prepare.run_wall_time_us":1612660,"raft.queue_time_us":1647,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":162,"replication_time_us":1101455,"row_lock_wait_count":1,"row_lock_wait_us":1611322,"thread_start_us":70,"threads_started":1}]]}
W20260812 08:01:29.665591 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 987ms of latency in SegmentAllocator::Sync()
W20260812 08:01:29.766482 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:29.990509 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 79) took 1076 ms (client timeout 50 ms). Trace:
I20260812 08:01:29.990509 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.990736 32089 rpcz_store.cc:269] 0812 08:01:28.914052 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.914202 (+   150us) service_pool.cc:224] Handling call
0812 08:01:28.914501 (+   299us) raft_consensus.cc:1471] Updating replica for 11 ops
0812 08:01:28.914762 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:28.915049 (+   287us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:28.915150 (+   101us) write_op.cc:270] Start()
0812 08:01:28.915181 (+    31us) write_op.cc:276] Timestamp: P: 1786521688559851 usec, L: 0
0812 08:01:28.915557 (+   376us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:28.915607 (+    50us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:28.915613 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:28.915630 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:29.990351 (+1074721us) raft_consensus.cc:1685] finished
0812 08:01:29.990354 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.990478 (+   124us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:28.915495 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:28.915631 (+   136us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:28.915636 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:28.915639 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:28.915898 (+   259us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:28.915908 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:28.915911 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:28.917339 (+  1428us) tablet.cc:701] Row locks acquired
0812 08:01:28.917341 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":208,"prepare.run_cpu_time_us":903,"prepare.run_wall_time_us":1962,"row_lock_wait_count":1,"row_lock_wait_us":1137,"thread_start_us":88,"threads_started":1}]]}
I20260812 08:01:29.991003 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.991151 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 80) took 1025 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.991334 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 81) took 691 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.991240 32088 rpcz_store.cc:269] 0812 08:01:28.965300 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.965404 (+   104us) service_pool.cc:224] Handling call
0812 08:01:29.990453 (+1025049us) raft_consensus.cc:1471] Updating replica for 11 ops
0812 08:01:29.990855 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.990864 (+     9us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.990875 (+    11us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.990878 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.990890 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.991054 (+   164us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:29.991093 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2356693888}
I20260812 08:01:29.991261 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.16]   Dedup: 1.16->[]
W20260812 08:01:29.991398 32086 rpcz_store.cc:269] 0812 08:01:29.299581 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.299844 (+   263us) service_pool.cc:224] Handling call
0812 08:01:29.990960 (+691116us) raft_consensus.cc:1471] Updating replica for 11 ops
0812 08:01:29.991118 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.991121 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.991151 (+    30us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.991155 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.991169 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.991292 (+   123us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:29.991321 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588761856}
W20260812 08:01:29.991787 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 82) took 641 ms (client timeout 50 ms). Trace:
W20260812 08:01:29.991863 32090 rpcz_store.cc:269] 0812 08:01:29.350526 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.350641 (+   115us) service_pool.cc:224] Handling call
0812 08:01:29.991224 (+640583us) raft_consensus.cc:1471] Updating replica for 11 ops
0812 08:01:29.991578 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 15
0812 08:01:29.991582 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:29.991590 (+     8us) raft_consensus.cc:1645] Marking committed up to 15
0812 08:01:29.991594 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.991605 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:29.991740 (+   135us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:29.991775 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1472498432}
I20260812 08:01:29.992024 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.16->[1.17-1.17]
W20260812 08:01:29.993145 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 988ms of latency in SegmentAllocator::Sync()
W20260812 08:01:30.653313 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 97) took 988 ms (client timeout 50 ms). Trace:
I20260812 08:01:30.653354 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.653432 31936 rpcz_store.cc:269] 0812 08:01:29.664437 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.664519 (+    82us) service_pool.cc:224] Handling call
0812 08:01:29.664669 (+   150us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:29.664779 (+   110us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:29.664781 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:29.664919 (+   138us) write_op.cc:270] Start()
0812 08:01:29.664947 (+    28us) write_op.cc:276] Timestamp: P: 1786521689663579 usec, L: 0
0812 08:01:29.665332 (+   385us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:29.665372 (+    40us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:29.665378 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.665391 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:30.653162 (+987771us) raft_consensus.cc:1685] finished
0812 08:01:30.653165 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.653274 (+   109us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:29.665267 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:29.665410 (+   143us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:29.665414 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:29.665416 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:29.665740 (+   324us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:29.665753 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:29.665756 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:29.666013 (+   257us) tablet.cc:701] Row locks acquired
0812 08:01:29.666015 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":231,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":801,"thread_start_us":116,"threads_started":1}]]}
I20260812 08:01:30.653860 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.653936 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 100) took 492 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.654047 31939 rpcz_store.cc:269] 0812 08:01:30.161627 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.161755 (+   128us) service_pool.cc:224] Handling call
0812 08:01:30.653307 (+491552us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:30.653637 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.653641 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.653652 (+    11us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.653655 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.653694 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.653866 (+   172us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:30.653919 (+    53us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129965824}
I20260812 08:01:30.654275 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.654332 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 98) took 938 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.654388 31938 rpcz_store.cc:269] 0812 08:01:29.716110 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.716229 (+   119us) service_pool.cc:224] Handling call
0812 08:01:30.653826 (+937597us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:30.654061 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.654065 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.654133 (+    68us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.654138 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.654152 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.654283 (+   131us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:30.654317 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2155807488}
W20260812 08:01:30.654721 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 99) took 544 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.654780 31941 rpcz_store.cc:269] 0812 08:01:30.110671 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.110810 (+   139us) service_pool.cc:224] Handling call
0812 08:01:30.654238 (+543428us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:30.654387 (+   149us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.654393 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.654403 (+    10us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.654408 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.654424 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.654546 (+   122us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:30.654704 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1249356544}
W20260812 08:01:30.664913 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19080
I20260812 08:01:30.734879 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17]   Dedup: 1.17->[]
I20260812 08:01:30.758445 32486 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=13, attempt_no=0}) took 17465 ms. Trace:
W20260812 08:01:30.758843 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1011ms of latency in SegmentAllocator::Sync()
I20260812 08:01:30.758652 32486 rpcz_store.cc:276] 0812 08:01:13.293227 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.293934 (+   707us) service_pool.cc:224] Handling call
0812 08:01:30.758424 (+17464490us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:28.560206 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:28.560324 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:28.560328 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:28.560330 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:28.560615 (+   285us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:28.560623 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:28.560626 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:29.663584 (+1102958us) lock_manager.cc:598] Waited 1102640us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:29.663603 (+    19us) tablet.cc:701] Row locks acquired
0812 08:01:29.663606 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:01:29.663653 (+    47us) write_op.cc:270] Start()
0812 08:01:29.663674 (+    21us) write_op.cc:276] Timestamp: P: 1786521689663579 usec, L: 0
0812 08:01:29.663676 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:29.663866 (+   190us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:30.755798 (+1091932us) op_driver.cc:464] REPLICATION: finished
0812 08:01:30.756147 (+   349us) write_op.cc:301] APPLY: starting
0812 08:01:30.756194 (+    47us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:30.756250 (+    56us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:30.756251 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:30.757997 (+  1746us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:30.758051 (+    54us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:30.758057 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:30.758203 (+   146us) log.cc:844] Serialized 319 byte log entry
0812 08:01:30.758303 (+   100us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:30.758333 (+    30us) write_op.cc:454] Released schema lock
0812 08:01:30.758342 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":270,"num_ops":50,"prepare.queue_time_us":15266050,"prepare.run_cpu_time_us":1394,"prepare.run_wall_time_us":1103753,"raft.queue_time_us":112,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":216,"replication_time_us":1092069,"row_lock_wait_count":1,"row_lock_wait_us":1102640,"thread_start_us":105,"threads_started":1}]]}
W20260812 08:01:30.760516 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 945ms of latency in SegmentAllocator::Sync()
W20260812 08:01:30.860862 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:30.981864 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.981916 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 84) took 1192 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.982093 32085 rpcz_store.cc:269] 0812 08:01:29.789743 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.789843 (+   100us) service_pool.cc:224] Handling call
0812 08:01:29.991946 (+202103us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:29.992152 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:29.992371 (+   219us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:29.992495 (+   124us) write_op.cc:270] Start()
0812 08:01:29.992522 (+    27us) write_op.cc:276] Timestamp: P: 1786521689663579 usec, L: 0
0812 08:01:29.992877 (+   355us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:29.992919 (+    42us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:29.992925 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:29.992937 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:30.981585 (+988648us) raft_consensus.cc:1685] finished
0812 08:01:30.981587 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.981846 (+   259us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.981897 (+    51us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:29.992765 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:29.992894 (+   129us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:29.992901 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:29.992902 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:29.993164 (+   262us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:29.993174 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:29.993177 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:29.994230 (+  1053us) tablet.cc:701] Row locks acquired
0812 08:01:29.994232 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":463771904,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":822,"prepare.run_wall_time_us":1517,"row_lock_wait_count":1,"row_lock_wait_us":740,"thread_start_us":83,"threads_started":1}]]}
I20260812 08:01:30.982288 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.982344 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 83) took 1244 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.982393 32087 rpcz_store.cc:269] 0812 08:01:29.738395 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.738516 (+   121us) service_pool.cc:224] Handling call
0812 08:01:30.981807 (+1243291us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:30.982092 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.982098 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.982114 (+    16us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.982123 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.982136 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.982282 (+   146us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.982328 (+    46us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2858360064}
I20260812 08:01:30.982604 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.982610 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 86) took 665 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.982736 32086 rpcz_store.cc:269] 0812 08:01:30.317066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.317206 (+   140us) service_pool.cc:224] Handling call
0812 08:01:30.982209 (+665003us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:30.982393 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.982397 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.982407 (+    10us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.982411 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.982421 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.982536 (+   115us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.982563 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1528607104}
I20260812 08:01:30.982847 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.982898 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 87) took 253 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.982947 32088 rpcz_store.cc:269] 0812 08:01:30.729542 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.729623 (+    81us) service_pool.cc:224] Handling call
0812 08:01:30.982545 (+252922us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:30.982722 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.982726 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.982733 (+     7us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.982737 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.982752 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.982861 (+   109us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.982887 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580951424}
I20260812 08:01:30.983073 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.17]   Dedup: 1.17->[]
W20260812 08:01:30.983176 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 88) took 202 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.983240 32089 rpcz_store.cc:269] 0812 08:01:30.780902 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.780988 (+    86us) service_pool.cc:224] Handling call
0812 08:01:30.982818 (+201830us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:30.982945 (+   127us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.982950 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.982960 (+    10us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.982966 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.982981 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.983116 (+   135us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.983163 (+    47us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":463211904}
W20260812 08:01:30.983319 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 85) took 717 ms (client timeout 50 ms). Trace:
W20260812 08:01:30.983373 32090 rpcz_store.cc:269] 0812 08:01:30.265881 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.266003 (+   122us) service_pool.cc:224] Handling call
0812 08:01:30.983046 (+717043us) raft_consensus.cc:1471] Updating replica for 12 ops
0812 08:01:30.983153 (+   107us) raft_consensus.cc:1534] Early marking committed up to index 16
0812 08:01:30.983155 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:30.983162 (+     7us) raft_consensus.cc:1645] Marking committed up to 16
0812 08:01:30.983166 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.983175 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:30.983277 (+   102us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:30.983306 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648339328}
I20260812 08:01:31.354203 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.18]   Dedup: 1.17->[1.18-1.18]
W20260812 08:01:31.355579 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 945ms of latency in SegmentAllocator::Sync()
W20260812 08:01:31.706411 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 103) took 946 ms (client timeout 50 ms). Trace:
I20260812 08:01:31.706451 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.17->[1.18-1.18]   Dedup: 1.18->[]
W20260812 08:01:31.706559 31941 rpcz_store.cc:269] 0812 08:01:30.759471 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.759538 (+    67us) service_pool.cc:224] Handling call
0812 08:01:30.759713 (+   175us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:30.759798 (+    85us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:30.759801 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:30.759924 (+   123us) write_op.cc:270] Start()
0812 08:01:30.759951 (+    27us) write_op.cc:276] Timestamp: P: 1786521690758416 usec, L: 0
0812 08:01:30.760283 (+   332us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:30.760323 (+    40us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:30.760328 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:30.760341 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:31.706215 (+945874us) raft_consensus.cc:1685] finished
0812 08:01:31.706217 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:31.706381 (+   164us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:30.760254 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:30.760395 (+   141us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:30.760400 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:30.760402 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:30.760832 (+   430us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:30.760841 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:30.760844 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:30.761033 (+   189us) tablet.cc:701] Row locks acquired
0812 08:01:30.761035 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":219,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":826,"thread_start_us":103,"threads_started":1}]]}
I20260812 08:01:31.707077 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.17->[1.18-1.18]   Dedup: 1.18->[]
W20260812 08:01:31.707141 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 104) took 896 ms (client timeout 50 ms). Trace:
W20260812 08:01:31.707193 31938 rpcz_store.cc:269] 0812 08:01:30.810686 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.810845 (+   159us) service_pool.cc:224] Handling call
0812 08:01:31.706402 (+895557us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:31.706888 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 17
0812 08:01:31.706911 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:31.706927 (+    16us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:31.706933 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:31.706953 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:31.707083 (+   130us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:31.707126 (+    43us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059108096}
I20260812 08:01:31.707341 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.17->[1.18-1.18]   Dedup: 1.18->[]
W20260812 08:01:31.707477 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 106) took 271 ms (client timeout 50 ms). Trace:
W20260812 08:01:31.707540 31936 rpcz_store.cc:269] 0812 08:01:31.435637 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.435723 (+    86us) service_pool.cc:224] Handling call
0812 08:01:31.707046 (+271323us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:31.707200 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 17
0812 08:01:31.707207 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:31.707215 (+     8us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:31.707219 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:31.707229 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:31.707410 (+   181us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:31.707458 (+    48us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623546112}
W20260812 08:01:31.707670 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 105) took 322 ms (client timeout 50 ms). Trace:
W20260812 08:01:31.707747 31939 rpcz_store.cc:269] 0812 08:01:31.384903 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.384995 (+    92us) service_pool.cc:224] Handling call
0812 08:01:31.707309 (+322314us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:31.707468 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 17
0812 08:01:31.707473 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:31.707484 (+    11us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:31.707491 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:31.707506 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:31.707622 (+   116us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:31.707658 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":740693760}
W20260812 08:01:31.759598 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19340
I20260812 08:01:31.811933 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.17->[1.18-1.18]   Dedup: 1.18->[]
I20260812 08:01:31.815101 32497 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=16, attempt_no=0}) took 18520 ms. Trace:
I20260812 08:01:31.815265 32497 rpcz_store.cc:276] 0812 08:01:13.294217 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.294261 (+    44us) service_pool.cc:224] Handling call
0812 08:01:31.815079 (+18520818us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:29.663997 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:29.664114 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:29.664117 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:29.664118 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:29.664426 (+   308us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:29.664433 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:29.664436 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:30.758365 (+1093929us) lock_manager.cc:598] Waited 1093609us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:30.758381 (+    16us) tablet.cc:701] Row locks acquired
0812 08:01:30.758383 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:30.758445 (+    62us) write_op.cc:270] Start()
0812 08:01:30.758470 (+    25us) write_op.cc:276] Timestamp: P: 1786521690758416 usec, L: 0
0812 08:01:30.758472 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:30.758656 (+   184us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:31.812939 (+1054283us) op_driver.cc:464] REPLICATION: finished
0812 08:01:31.813239 (+   300us) write_op.cc:301] APPLY: starting
0812 08:01:31.813280 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:31.813381 (+   101us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:31.813384 (+     3us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:31.814734 (+  1350us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:31.814777 (+    43us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:31.814781 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:31.814883 (+   102us) log.cc:844] Serialized 319 byte log entry
0812 08:01:31.814964 (+    81us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:31.814992 (+    28us) write_op.cc:454] Released schema lock
0812 08:01:31.815000 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":236,"num_ops":50,"prepare.queue_time_us":16369511,"prepare.run_cpu_time_us":1409,"prepare.run_wall_time_us":1094734,"raft.queue_time_us":273,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":357,"replication_time_us":1054424,"row_lock_wait_count":1,"row_lock_wait_us":1093609,"spinlock_wait_cycles":23296,"thread_start_us":153,"threads_started":1}]]}
W20260812 08:01:31.815594 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 952ms of latency in SegmentAllocator::Sync()
W20260812 08:01:31.817952 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1000ms of latency in SegmentAllocator::Sync()
W20260812 08:01:31.918090 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:32.078846 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 31: this message will repeat every 5th retry.
I20260812 08:01:32.301186 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.18]   Dedup: 1.18->[]
W20260812 08:01:32.301186 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 89) took 947 ms (client timeout 50 ms). Trace:
W20260812 08:01:32.301421 32090 rpcz_store.cc:269] 0812 08:01:31.353705 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.353831 (+   126us) service_pool.cc:224] Handling call
0812 08:01:31.354135 (+   304us) raft_consensus.cc:1471] Updating replica for 13 ops
0812 08:01:31.354421 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 17
0812 08:01:31.354663 (+   242us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:31.354914 (+   251us) write_op.cc:270] Start()
0812 08:01:31.354944 (+    30us) write_op.cc:276] Timestamp: P: 1786521690758416 usec, L: 0
0812 08:01:31.355294 (+   350us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:31.355340 (+    46us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:31.355346 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:31.355360 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:32.301021 (+945661us) raft_consensus.cc:1685] finished
0812 08:01:32.301023 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.301162 (+   139us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:31.355254 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:31.355406 (+   152us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:31.355412 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:31.355413 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:31.355825 (+   412us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:31.355837 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:31.355840 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:31.356407 (+   567us) tablet.cc:701] Row locks acquired
0812 08:01:31.356416 (+     9us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":211,"prepare.run_cpu_time_us":931,"prepare.run_wall_time_us":1217,"row_lock_wait_count":1,"row_lock_wait_us":354,"thread_start_us":83,"threads_started":1}]]}
W20260812 08:01:32.301620 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 90) took 896 ms (client timeout 50 ms). Trace:
I20260812 08:01:32.301640 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.19]   Dedup: 1.18->[1.19-1.19]
W20260812 08:01:32.301719 32089 rpcz_store.cc:269] 0812 08:01:31.405331 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.405406 (+    75us) service_pool.cc:224] Handling call
0812 08:01:32.301131 (+895725us) raft_consensus.cc:1471] Updating replica for 13 ops
0812 08:01:32.301373 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 17
0812 08:01:32.301376 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:32.301386 (+    10us) raft_consensus.cc:1645] Marking committed up to 17
0812 08:01:32.301389 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.301399 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.301564 (+   165us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:32.301604 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059265920}
W20260812 08:01:32.302861 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 986ms of latency in SegmentAllocator::Sync()
W20260812 08:01:32.816676 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19600
W20260812 08:01:32.818538 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 109) took 1001 ms (client timeout 50 ms). Trace:
I20260812 08:01:32.818543 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19]   Dedup: 1.19->[]
W20260812 08:01:32.818729 31939 rpcz_store.cc:269] 0812 08:01:31.816726 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.816775 (+    49us) service_pool.cc:224] Handling call
0812 08:01:31.817111 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:31.817241 (+   130us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:31.817244 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:31.817362 (+   118us) write_op.cc:270] Start()
0812 08:01:31.817384 (+    22us) write_op.cc:276] Timestamp: P: 1786521691815133 usec, L: 0
0812 08:01:31.817744 (+   360us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:31.818243 (+   499us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:31.818252 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:31.818268 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:32.818390 (+1000122us) raft_consensus.cc:1685] finished
0812 08:01:32.818393 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.818496 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:31.817734 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:31.817857 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:31.817863 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:31.817867 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:31.818243 (+   376us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:31.818257 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:31.818260 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:31.818489 (+   229us) tablet.cc:701] Row locks acquired
0812 08:01:31.818491 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":209,"prepare.run_cpu_time_us":809,"prepare.run_wall_time_us":809,"thread_start_us":128,"threads_started":1}]]}
I20260812 08:01:32.819222 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19]   Dedup: 1.19->[]
W20260812 08:01:32.819288 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 111) took 425 ms (client timeout 50 ms). Trace:
W20260812 08:01:32.819332 31938 rpcz_store.cc:269] 0812 08:01:32.393380 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.393456 (+    76us) service_pool.cc:224] Handling call
0812 08:01:32.818494 (+425038us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:32.819032 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:32.819036 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:32.819051 (+    15us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:32.819057 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.819091 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.819234 (+   143us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:32.819274 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":977019648}
W20260812 08:01:32.819487 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 112) took 375 ms (client timeout 50 ms). Trace:
I20260812 08:01:32.819557 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19]   Dedup: 1.19->[]
W20260812 08:01:32.819525 31941 rpcz_store.cc:269] 0812 08:01:32.444282 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.444389 (+   107us) service_pool.cc:224] Handling call
0812 08:01:32.819177 (+374788us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:32.819333 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:32.819336 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:32.819347 (+    11us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:32.819350 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.819359 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.819458 (+    99us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:32.819478 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861493120}
W20260812 08:01:32.819890 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 110) took 952 ms (client timeout 50 ms). Trace:
W20260812 08:01:32.819949 31936 rpcz_store.cc:269] 0812 08:01:31.867926 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.868041 (+   115us) service_pool.cc:224] Handling call
0812 08:01:32.819510 (+951469us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:32.819706 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:32.819709 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:32.819719 (+    10us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:32.819724 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.819737 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:32.819850 (+   113us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:32.819878 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2187694848}
I20260812 08:01:32.849364 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19]   Dedup: 1.19->[]
I20260812 08:01:32.852829 32504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=17, attempt_no=0}) took 19557 ms. Trace:
I20260812 08:01:32.852946 32504 rpcz_store.cc:276] 0812 08:01:13.295764 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.295812 (+    48us) service_pool.cc:224] Handling call
0812 08:01:32.852808 (+19556996us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:30.758756 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:30.758868 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:30.758872 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:30.758873 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:30.759153 (+   280us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:30.759161 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:30.759163 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:31.815125 (+1055962us) lock_manager.cc:598] Waited 1055693us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:31.815147 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:31.815148 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:31.815186 (+    38us) write_op.cc:270] Start()
0812 08:01:31.815213 (+    27us) write_op.cc:276] Timestamp: P: 1786521691815133 usec, L: 0
0812 08:01:31.815215 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:31.815357 (+   142us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:32.850341 (+1034984us) op_driver.cc:464] REPLICATION: finished
0812 08:01:32.850699 (+   358us) write_op.cc:301] APPLY: starting
0812 08:01:32.850733 (+    34us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:32.850810 (+    77us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:32.850812 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:32.852410 (+  1598us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:32.852471 (+    61us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:32.852477 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:32.852611 (+   134us) log.cc:844] Serialized 319 byte log entry
0812 08:01:32.852683 (+    72us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:32.852717 (+    34us) write_op.cc:454] Released schema lock
0812 08:01:32.852726 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":296,"num_ops":50,"prepare.queue_time_us":17462706,"prepare.run_cpu_time_us":1202,"prepare.run_wall_time_us":1056686,"raft.queue_time_us":1412,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":199,"replication_time_us":1035086,"row_lock_wait_count":1,"row_lock_wait_us":1055693,"thread_start_us":122,"threads_started":1}]]}
W20260812 08:01:32.853770 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 963ms of latency in SegmentAllocator::Sync()
W20260812 08:01:32.855321 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1192ms of latency in SegmentAllocator::Sync()
W20260812 08:01:32.955273 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:33.041899 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 19 similar messages]
I20260812 08:01:33.289467 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.19]   Dedup: 1.19->[]
W20260812 08:01:33.289597 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 92) took 1260 ms (client timeout 50 ms). Trace:
I20260812 08:01:33.289829 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.19]   Dedup: 1.19->[]
W20260812 08:01:33.289847 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 94) took 672 ms (client timeout 50 ms). Trace:
W20260812 08:01:33.289991 32090 rpcz_store.cc:269] 0812 08:01:32.617143 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.617220 (+    77us) service_pool.cc:224] Handling call
0812 08:01:33.289414 (+672194us) raft_consensus.cc:1471] Updating replica for 14 ops
0812 08:01:33.289656 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:33.289659 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:33.289670 (+    11us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:33.289674 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:33.289690 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:33.289800 (+   110us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:33.289833 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1544745728}
W20260812 08:01:33.289666 32086 rpcz_store.cc:269] 0812 08:01:32.028811 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.028933 (+   122us) service_pool.cc:224] Handling call
0812 08:01:32.301565 (+272632us) raft_consensus.cc:1471] Updating replica for 14 ops
0812 08:01:32.301856 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:32.302073 (+   217us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:32.302221 (+   148us) write_op.cc:270] Start()
0812 08:01:32.302248 (+    27us) write_op.cc:276] Timestamp: P: 1786521691815133 usec, L: 0
0812 08:01:32.302571 (+   323us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:32.302642 (+    71us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:32.302648 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.302661 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:33.289304 (+986643us) raft_consensus.cc:1685] finished
0812 08:01:33.289307 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:33.289516 (+   209us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:33.289576 (+    60us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:32.302654 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:32.302855 (+   201us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:32.302860 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:32.302862 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:32.303193 (+   331us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:32.303203 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:32.303206 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:32.303824 (+   618us) tablet.cc:701] Row locks acquired
0812 08:01:32.303826 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":626111232,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":247,"prepare.run_cpu_time_us":934,"prepare.run_wall_time_us":1287,"row_lock_wait_count":1,"row_lock_wait_us":411,"thread_start_us":74,"threads_started":1}]]}
W20260812 08:01:33.290158 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 93) took 724 ms (client timeout 50 ms). Trace:
W20260812 08:01:33.290416 32089 rpcz_store.cc:269] 0812 08:01:32.565963 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.566090 (+   127us) service_pool.cc:224] Handling call
0812 08:01:33.289801 (+723711us) raft_consensus.cc:1471] Updating replica for 14 ops
0812 08:01:33.289976 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:33.289979 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:33.289985 (+     6us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:33.290000 (+    15us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:33.290017 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:33.290122 (+   105us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:33.290147 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1663659904}
I20260812 08:01:33.290158 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.19->[1.20-1.20]
W20260812 08:01:33.291883 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 935ms of latency in SegmentAllocator::Sync()
W20260812 08:01:33.854081 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d198c0
W20260812 08:01:34.047981 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 115) took 1193 ms (client timeout 50 ms). Trace:
I20260812 08:01:34.048044 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.19->[1.20-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.048112 31936 rpcz_store.cc:269] 0812 08:01:32.854061 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.854167 (+   106us) service_pool.cc:224] Handling call
0812 08:01:32.854304 (+   137us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:32.854372 (+    68us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:32.854374 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:32.854485 (+   111us) write_op.cc:270] Start()
0812 08:01:32.854506 (+    21us) write_op.cc:276] Timestamp: P: 1786521692852778 usec, L: 0
0812 08:01:32.855029 (+   523us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:32.855104 (+    75us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:32.855112 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:32.855126 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:34.047852 (+1192726us) raft_consensus.cc:1685] finished
0812 08:01:34.047854 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.047953 (+    99us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:32.854854 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:32.854962 (+   108us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:32.854966 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:32.854968 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:32.855307 (+   339us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:32.855318 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:32.855321 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:32.855601 (+   280us) tablet.cc:701] Row locks acquired
0812 08:01:32.855602 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":270,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":802,"thread_start_us":158,"threads_started":1}]]}
I20260812 08:01:34.048467 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.19->[1.20-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.048606 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 117) took 664 ms (client timeout 50 ms). Trace:
I20260812 08:01:34.048745 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.19->[1.20-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.048781 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 116) took 1143 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.048676 31938 rpcz_store.cc:269] 0812 08:01:33.384233 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.384323 (+    90us) service_pool.cc:224] Handling call
0812 08:01:34.047995 (+663672us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:34.048328 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.048337 (+     9us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.048348 (+    11us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.048352 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.048363 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.048539 (+   176us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:34.048592 (+    53us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525810944}
W20260812 08:01:34.048933 31941 rpcz_store.cc:269] 0812 08:01:32.904850 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.904948 (+    98us) service_pool.cc:224] Handling call
0812 08:01:34.048437 (+1143489us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:34.048599 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.048601 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.048609 (+     8us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.048615 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.048631 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.048744 (+   113us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:34.048770 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2629287168}
I20260812 08:01:34.049073 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.19->[1.20-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.049072 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 119) took 60 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.049216 31935 rpcz_store.cc:269] 0812 08:01:33.988442 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.988550 (+   108us) service_pool.cc:224] Handling call
0812 08:01:34.048710 (+ 60160us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:34.048906 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.048911 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.048918 (+     7us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.048922 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.048931 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.049036 (+   105us) spinlock_profiling.cc:243] Waited 59.9 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:34.049062 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137772288}
I20260812 08:01:34.049486 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.19->[1.20-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.049755 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 118) took 614 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.049839 31939 rpcz_store.cc:269] 0812 08:01:33.435294 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.435381 (+    87us) service_pool.cc:224] Handling call
0812 08:01:34.049451 (+614070us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:34.049602 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.049606 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.049616 (+    10us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.049623 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.049634 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.049724 (+    90us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:34.049745 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1411822976}
I20260812 08:01:34.052867 32514 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=18, attempt_no=0}) took 20756 ms. Trace:
I20260812 08:01:34.053059 32514 rpcz_store.cc:276] 0812 08:01:13.296247 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.296336 (+    89us) service_pool.cc:224] Handling call
0812 08:01:34.052844 (+20756508us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:31.815475 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:31.815602 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:31.815621 (+    19us) write_op.cc:435] Acquired schema lock
0812 08:01:31.815624 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:31.816202 (+   578us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:31.816212 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:31.816216 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:32.852747 (+1036531us) lock_manager.cc:598] Waited 1036170us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:32.852761 (+    14us) tablet.cc:701] Row locks acquired
0812 08:01:32.852763 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:32.852830 (+    67us) write_op.cc:270] Start()
0812 08:01:32.852861 (+    31us) write_op.cc:276] Timestamp: P: 1786521692852778 usec, L: 0
0812 08:01:32.852864 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:32.853040 (+   176us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:34.050287 (+1197247us) op_driver.cc:464] REPLICATION: finished
0812 08:01:34.050504 (+   217us) write_op.cc:301] APPLY: starting
0812 08:01:34.050543 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:34.050647 (+   104us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:34.050648 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:34.052410 (+  1762us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:34.052470 (+    60us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:34.052477 (+     7us) write_op.cc:312] APPLY: finished
0812 08:01:34.052634 (+   157us) log.cc:844] Serialized 319 byte log entry
0812 08:01:34.052713 (+    79us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:34.052746 (+    33us) write_op.cc:454] Released schema lock
0812 08:01:34.052756 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":151,"num_ops":50,"prepare.queue_time_us":18518877,"prepare.run_cpu_time_us":1733,"prepare.run_wall_time_us":1037661,"raft.queue_time_us":105,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":460,"replication_time_us":1197367,"row_lock_wait_count":1,"row_lock_wait_us":1036170,"spinlock_wait_cycles":32256,"thread_start_us":82,"threads_started":1}]]}
W20260812 08:01:34.053449 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 959ms of latency in SegmentAllocator::Sync()
W20260812 08:01:34.055325 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 952ms of latency in SegmentAllocator::Sync()
W20260812 08:01:34.155207 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:34.227483 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.227582 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 95) took 1235 ms (client timeout 50 ms). Trace:
I20260812 08:01:34.227825 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.227686 32087 rpcz_store.cc:269] 0812 08:01:32.991953 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.992058 (+   105us) service_pool.cc:224] Handling call
0812 08:01:33.290074 (+298016us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:33.290866 (+   792us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:33.291098 (+   232us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:33.291239 (+   141us) write_op.cc:270] Start()
0812 08:01:33.291276 (+    37us) write_op.cc:276] Timestamp: P: 1786521692852778 usec, L: 0
0812 08:01:33.291613 (+   337us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:33.291668 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:33.291674 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:33.291685 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:34.227304 (+935619us) raft_consensus.cc:1685] finished
0812 08:01:34.227307 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.227502 (+   195us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.227564 (+    62us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:33.291529 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:33.291672 (+   143us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:33.291677 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:33.291679 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:33.291999 (+   320us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:33.292017 (+    18us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:33.292023 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:33.292893 (+   870us) tablet.cc:701] Row locks acquired
0812 08:01:33.292895 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":684566400,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":170,"prepare.run_cpu_time_us":942,"prepare.run_wall_time_us":1415,"row_lock_wait_count":1,"row_lock_wait_us":640,"thread_start_us":92,"threads_started":1}]]}
W20260812 08:01:34.227965 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 96) took 1184 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.228099 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 98) took 627 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.228039 32085 rpcz_store.cc:269] 0812 08:01:33.043427 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.043724 (+   297us) service_pool.cc:224] Handling call
0812 08:01:34.227421 (+1183697us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:34.227641 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.227644 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.227655 (+    11us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.227658 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.227668 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.227877 (+   209us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.227946 (+    69us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2721404800}
W20260812 08:01:34.228184 32086 rpcz_store.cc:269] 0812 08:01:33.601144 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.601282 (+   138us) service_pool.cc:224] Handling call
0812 08:01:34.227791 (+626509us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:34.227923 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.227926 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.227933 (+     7us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.227937 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.227950 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.228060 (+   110us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.228087 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1439768448}
I20260812 08:01:34.228185 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.20->[]
I20260812 08:01:34.228554 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.19]   Dedup: 1.19->[]
W20260812 08:01:34.228647 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 97) took 678 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.228717 32089 rpcz_store.cc:269] 0812 08:01:33.549968 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:33.550070 (+   102us) service_pool.cc:224] Handling call
0812 08:01:34.228153 (+678083us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:34.228438 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.228441 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.228449 (+     8us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.228454 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.228466 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.228591 (+   125us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.228614 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1558627072}
I20260812 08:01:34.228818 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.228868 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 91) took 2251 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.228909 32088 rpcz_store.cc:269] 0812 08:01:31.977677 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.977795 (+   118us) service_pool.cc:224] Handling call
0812 08:01:34.228530 (+2250735us) raft_consensus.cc:1471] Updating replica for 14 ops
0812 08:01:34.228668 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 18
0812 08:01:34.228670 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.228696 (+    26us) raft_consensus.cc:1645] Marking committed up to 18
0812 08:01:34.228703 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.228716 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.228828 (+   112us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.228857 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5175522560}
I20260812 08:01:34.229018 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.20]   Dedup: 1.20->[]
W20260812 08:01:34.229058 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 100) took 171 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.229106 32084 rpcz_store.cc:269] 0812 08:01:34.057836 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.057949 (+   113us) service_pool.cc:224] Handling call
0812 08:01:34.228760 (+170811us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:34.228908 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.228911 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.228919 (+     8us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.228923 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.228931 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.229028 (+    97us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.229047 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391815040}
W20260812 08:01:34.229313 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 99) took 222 ms (client timeout 50 ms). Trace:
W20260812 08:01:34.229387 32090 rpcz_store.cc:269] 0812 08:01:34.006945 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.007030 (+    85us) service_pool.cc:224] Handling call
0812 08:01:34.228984 (+221954us) raft_consensus.cc:1471] Updating replica for 15 ops
0812 08:01:34.229113 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 19
0812 08:01:34.229119 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:34.229129 (+    10us) raft_consensus.cc:1645] Marking committed up to 19
0812 08:01:34.229133 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.229143 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:34.229253 (+   110us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:34.229298 (+    45us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":509658496}
I20260812 08:01:34.491457 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.21]   Dedup: 1.20->[1.21-1.21]
W20260812 08:01:34.492676 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1025ms of latency in SegmentAllocator::Sync()
W20260812 08:01:34.591976 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 36: this message will repeat every 5th retry.
I20260812 08:01:35.007907 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.20->[1.21-1.21]   Dedup: 1.21->[]
W20260812 08:01:35.007936 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 122) took 954 ms (client timeout 50 ms). Trace:
I20260812 08:01:35.008291 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.20->[1.21-1.21]   Dedup: 1.21->[]
W20260812 08:01:35.008394 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 123) took 903 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.008179 31939 rpcz_store.cc:269] 0812 08:01:34.053746 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.054149 (+   403us) service_pool.cc:224] Handling call
0812 08:01:34.054331 (+   182us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:34.054421 (+    90us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:34.054424 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:34.054533 (+   109us) write_op.cc:270] Start()
0812 08:01:34.054554 (+    21us) write_op.cc:276] Timestamp: P: 1786521694052960 usec, L: 0
0812 08:01:34.055019 (+   465us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:34.055108 (+    89us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:34.055115 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.055129 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:35.007739 (+952610us) raft_consensus.cc:1685] finished
0812 08:01:35.007742 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.007908 (+   166us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:34.055101 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:34.055231 (+   130us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:34.055236 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:34.055238 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:34.055711 (+   473us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:34.055724 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:34.055728 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:34.056034 (+   306us) tablet.cc:701] Row locks acquired
0812 08:01:34.056036 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":363,"prepare.run_cpu_time_us":1072,"prepare.run_wall_time_us":1071,"thread_start_us":208,"threads_started":1}]]}
W20260812 08:01:35.008445 31935 rpcz_store.cc:269] 0812 08:01:34.105088 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.105182 (+    94us) service_pool.cc:224] Handling call
0812 08:01:35.007856 (+902674us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:35.008104 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.008106 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.008117 (+    11us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.008119 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.008129 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.008342 (+   213us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:35.008378 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2075374336}
I20260812 08:01:35.008555 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.20->[1.21-1.21]   Dedup: 1.21->[]
W20260812 08:01:35.008587 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 124) took 342 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.008630 31942 rpcz_store.cc:269] 0812 08:01:34.666533 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.666640 (+   107us) service_pool.cc:224] Handling call
0812 08:01:35.008258 (+341618us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:35.008423 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.008425 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.008435 (+    10us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.008442 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.008454 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.008553 (+    99us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:35.008579 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":785143552}
W20260812 08:01:35.008841 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 125) took 291 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.008921 31941 rpcz_store.cc:269] 0812 08:01:34.717371 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.717489 (+   118us) service_pool.cc:224] Handling call
0812 08:01:35.008523 (+291034us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:35.008654 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.008657 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.008665 (+     8us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.008669 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.008679 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.008795 (+   116us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:35.008826 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":668855552}
W20260812 08:01:35.054246 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19b80
I20260812 08:01:35.129189 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.20->[1.21-1.21]   Dedup: 1.21->[]
I20260812 08:01:35.132848 32525 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=19, attempt_no=0}) took 21836 ms. Trace:
I20260812 08:01:35.133023 32525 rpcz_store.cc:276] 0812 08:01:13.296745 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.296818 (+    73us) service_pool.cc:224] Handling call
0812 08:01:35.132825 (+21836007us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:32.853171 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:32.853287 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:32.853294 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:32.853297 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:32.853644 (+   347us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:32.853654 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:32.853658 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:34.052941 (+1199283us) lock_manager.cc:598] Waited 1199006us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:34.052956 (+    15us) tablet.cc:701] Row locks acquired
0812 08:01:34.052961 (+     5us) write_op.cc:260] PREPARE: finished
0812 08:01:34.053009 (+    48us) write_op.cc:270] Start()
0812 08:01:34.053038 (+    29us) write_op.cc:276] Timestamp: P: 1786521694052960 usec, L: 0
0812 08:01:34.053041 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:34.053206 (+   165us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:35.130324 (+1077118us) op_driver.cc:464] REPLICATION: finished
0812 08:01:35.130735 (+   411us) write_op.cc:301] APPLY: starting
0812 08:01:35.130777 (+    42us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:35.130940 (+   163us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:35.130942 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:35.132447 (+  1505us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:35.132489 (+    42us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:35.132493 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:35.132629 (+   136us) log.cc:844] Serialized 319 byte log entry
0812 08:01:35.132701 (+    72us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:35.132728 (+    27us) write_op.cc:454] Released schema lock
0812 08:01:35.132738 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":258,"num_ops":50,"prepare.queue_time_us":19556096,"prepare.run_cpu_time_us":1357,"prepare.run_wall_time_us":1200121,"raft.queue_time_us":33,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":176,"replication_time_us":1077196,"row_lock_wait_count":1,"row_lock_wait_us":1199006,"thread_start_us":97,"threads_started":1}]]}
W20260812 08:01:35.133340 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1059ms of latency in SegmentAllocator::Sync()
W20260812 08:01:35.135500 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 923ms of latency in SegmentAllocator::Sync()
W20260812 08:01:35.235430 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:35.518369 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 101) took 1027 ms (client timeout 50 ms). Trace:
I20260812 08:01:35.518369 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.21]   Dedup: 1.21->[]
I20260812 08:01:35.518854 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.21]   Dedup: 1.21->[]
W20260812 08:01:35.518644 32090 rpcz_store.cc:269] 0812 08:01:34.490994 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.491089 (+    95us) service_pool.cc:224] Handling call
0812 08:01:34.491385 (+   296us) raft_consensus.cc:1471] Updating replica for 16 ops
0812 08:01:34.491599 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:34.491852 (+   253us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:34.491980 (+   128us) write_op.cc:270] Start()
0812 08:01:34.492008 (+    28us) write_op.cc:276] Timestamp: P: 1786521694052960 usec, L: 0
0812 08:01:34.492320 (+   312us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:34.492363 (+    43us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:34.492369 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:34.492383 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:35.518179 (+1025796us) raft_consensus.cc:1685] finished
0812 08:01:35.518182 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.518332 (+   150us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:34.492243 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:34.492366 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:34.492370 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:34.492372 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:34.492585 (+   213us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:34.492594 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:34.492596 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:34.493573 (+   977us) tablet.cc:701] Row locks acquired
0812 08:01:34.493575 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":1383,"row_lock_wait_count":1,"row_lock_wait_us":750,"thread_start_us":72,"threads_started":1}]]}
I20260812 08:01:35.519163 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.21]   Dedup: 1.21->[]
W20260812 08:01:35.518900 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 102) took 977 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.519315 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 104) took 470 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.519474 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 103) took 522 ms (client timeout 50 ms). Trace:
W20260812 08:01:35.519412 32089 rpcz_store.cc:269] 0812 08:01:35.048635 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.048747 (+   112us) service_pool.cc:224] Handling call
0812 08:01:35.518806 (+470059us) raft_consensus.cc:1471] Updating replica for 16 ops
0812 08:01:35.519041 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.519044 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.519052 (+     8us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.519055 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.519065 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.519271 (+   206us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:35.519302 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080235392}
W20260812 08:01:35.519538 32088 rpcz_store.cc:269] 0812 08:01:34.997517 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.997620 (+   103us) service_pool.cc:224] Handling call
0812 08:01:35.519129 (+521509us) raft_consensus.cc:1471] Updating replica for 16 ops
0812 08:01:35.519285 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.519288 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.519295 (+     7us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.519299 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.519309 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.519441 (+   132us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:35.519461 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1198623104}
W20260812 08:01:35.519269 32084 rpcz_store.cc:269] 0812 08:01:34.541844 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.541947 (+   103us) service_pool.cc:224] Handling call
0812 08:01:35.518312 (+976365us) raft_consensus.cc:1471] Updating replica for 16 ops
0812 08:01:35.518712 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 20
0812 08:01:35.518716 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:35.518729 (+    13us) raft_consensus.cc:1645] Marking committed up to 20
0812 08:01:35.518733 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.518745 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:35.518851 (+   106us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:35.518883 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244597120}
I20260812 08:01:35.519459 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.22]   Dedup: 1.21->[1.22-1.22]
W20260812 08:01:35.521232 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 932ms of latency in SegmentAllocator::Sync()
W20260812 08:01:36.059129 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 128) took 924 ms (client timeout 50 ms). Trace:
I20260812 08:01:36.059136 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.059306 31941 rpcz_store.cc:269] 0812 08:01:35.134262 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.134413 (+   151us) service_pool.cc:224] Handling call
0812 08:01:35.134561 (+   148us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:35.134718 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:35.134721 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:35.134847 (+   126us) write_op.cc:270] Start()
0812 08:01:35.134868 (+    21us) write_op.cc:276] Timestamp: P: 1786521695132889 usec, L: 0
0812 08:01:35.135238 (+   370us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:35.135281 (+    43us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:35.135287 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.135300 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:36.058983 (+923683us) raft_consensus.cc:1685] finished
0812 08:01:36.058985 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.059096 (+   111us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:35.135163 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:35.135296 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:35.135300 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:35.135302 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:35.135574 (+   272us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:35.135612 (+    38us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:35.135615 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:35.135844 (+   229us) tablet.cc:701] Row locks acquired
0812 08:01:35.135845 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":221,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":731,"thread_start_us":144,"threads_started":1}]]}
I20260812 08:01:36.059561 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.059607 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 129) took 874 ms (client timeout 50 ms). Trace:
W20260812 08:01:36.059669 31942 rpcz_store.cc:269] 0812 08:01:35.185390 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.185519 (+   129us) service_pool.cc:224] Handling call
0812 08:01:36.059089 (+873570us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:36.059406 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.059412 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.059426 (+    14us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.059433 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.059448 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.059546 (+    98us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:36.059575 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2008513792}
W20260812 08:01:36.059863 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 130) took 341 ms (client timeout 50 ms). Trace:
I20260812 08:01:36.059939 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.059931 31935 rpcz_store.cc:269] 0812 08:01:35.718920 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.719194 (+   274us) service_pool.cc:224] Handling call
0812 08:01:36.059527 (+340333us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:36.059654 (+   127us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.059657 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.059665 (+     8us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.059669 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.059680 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.059821 (+   141us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:36.059850 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":782227328}
W20260812 08:01:36.060304 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 131) took 290 ms (client timeout 50 ms). Trace:
W20260812 08:01:36.060367 31939 rpcz_store.cc:269] 0812 08:01:35.770018 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.770161 (+   143us) service_pool.cc:224] Handling call
0812 08:01:36.059913 (+289752us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:36.060135 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.060138 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.060145 (+     7us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.060149 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.060162 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.060267 (+   105us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:36.060293 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665911168}
W20260812 08:01:36.134155 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0000
I20260812 08:01:36.265561 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22]   Dedup: 1.22->[]
I20260812 08:01:36.269095 32533 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=20, attempt_no=0}) took 22970 ms. Trace:
W20260812 08:01:36.269556 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1020ms of latency in SegmentAllocator::Sync()
I20260812 08:01:36.269323 32533 rpcz_store.cc:276] 0812 08:01:13.298120 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:13.298207 (+    87us) service_pool.cc:224] Handling call
0812 08:01:36.269073 (+22970866us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:34.053330 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:34.053448 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:34.053452 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:34.053454 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:34.053812 (+   358us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:34.053823 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:34.053828 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:35.132872 (+1079044us) lock_manager.cc:598] Waited 1078748us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:35.132893 (+    21us) tablet.cc:701] Row locks acquired
0812 08:01:35.132895 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:35.132939 (+    44us) write_op.cc:270] Start()
0812 08:01:35.132966 (+    27us) write_op.cc:276] Timestamp: P: 1786521695132889 usec, L: 0
0812 08:01:35.132970 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:35.133121 (+   151us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:36.266891 (+1133770us) op_driver.cc:464] REPLICATION: finished
0812 08:01:36.267227 (+   336us) write_op.cc:301] APPLY: starting
0812 08:01:36.267269 (+    42us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:36.267352 (+    83us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:36.267357 (+     5us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:36.268681 (+  1324us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:36.268729 (+    48us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:36.268734 (+     5us) write_op.cc:312] APPLY: finished
0812 08:01:36.268850 (+   116us) log.cc:844] Serialized 319 byte log entry
0812 08:01:36.268947 (+    97us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:36.268976 (+    29us) write_op.cc:454] Released schema lock
0812 08:01:36.268985 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":264,"num_ops":50,"prepare.queue_time_us":20754878,"prepare.run_cpu_time_us":1354,"prepare.run_wall_time_us":1079895,"raft.queue_time_us":1003,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"replication_time_us":1133876,"row_lock_wait_count":1,"row_lock_wait_us":1078748,"spinlock_wait_cycles":896,"thread_start_us":134,"threads_started":1}]]}
W20260812 08:01:36.272646 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1035ms of latency in SegmentAllocator::Sync()
W20260812 08:01:36.372288 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:36.453922 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.454010 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 105) took 1057 ms (client timeout 50 ms). Trace:
I20260812 08:01:36.454226 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.454370 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 107) took 521 ms (client timeout 50 ms). Trace:
W20260812 08:01:36.454141 32086 rpcz_store.cc:269] 0812 08:01:35.396623 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.396766 (+   143us) service_pool.cc:224] Handling call
0812 08:01:35.519371 (+122605us) raft_consensus.cc:1471] Updating replica for 17 ops
0812 08:01:35.520231 (+   860us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:35.520444 (+   213us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:35.520567 (+   123us) write_op.cc:270] Start()
0812 08:01:35.520594 (+    27us) write_op.cc:276] Timestamp: P: 1786521695132889 usec, L: 0
0812 08:01:35.520948 (+   354us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:35.520990 (+    42us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:35.520996 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:35.521008 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:36.453712 (+932704us) raft_consensus.cc:1685] finished
0812 08:01:36.453714 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.453927 (+   213us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:36.453984 (+    57us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:35.520838 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:35.520965 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:35.520971 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:35.520974 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:35.521241 (+   267us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:35.521251 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:35.521253 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:35.522319 (+  1066us) tablet.cc:701] Row locks acquired
0812 08:01:35.522321 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":281214720,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":816,"prepare.run_wall_time_us":1533,"row_lock_wait_count":1,"row_lock_wait_us":766,"thread_start_us":100,"threads_started":1}]]}
W20260812 08:01:36.454555 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 108) took 470 ms (client timeout 50 ms). Trace:
W20260812 08:01:36.454442 32084 rpcz_store.cc:269] 0812 08:01:35.933057 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.933151 (+    94us) service_pool.cc:224] Handling call
0812 08:01:36.453862 (+520711us) raft_consensus.cc:1471] Updating replica for 17 ops
0812 08:01:36.454095 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.454098 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.454109 (+    11us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.454113 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.454123 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.454317 (+   194us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:36.454357 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1196677376}
W20260812 08:01:36.454650 32088 rpcz_store.cc:269] 0812 08:01:35.984286 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.984364 (+    78us) service_pool.cc:224] Handling call
0812 08:01:36.454196 (+469832us) raft_consensus.cc:1471] Updating replica for 17 ops
0812 08:01:36.454368 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.454371 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.454380 (+     9us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.454388 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.454399 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.454516 (+   117us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:36.454542 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1079482240}
I20260812 08:01:36.454499 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.22]   Dedup: 1.22->[]
W20260812 08:01:36.455116 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 106) took 1007 ms (client timeout 50 ms). Trace:
W20260812 08:01:36.455295 32085 rpcz_store.cc:269] 0812 08:01:35.447466 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.447565 (+    99us) service_pool.cc:224] Handling call
0812 08:01:36.454455 (+1006890us) raft_consensus.cc:1471] Updating replica for 17 ops
0812 08:01:36.454938 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 21
0812 08:01:36.454941 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:36.454950 (+     9us) raft_consensus.cc:1645] Marking committed up to 21
0812 08:01:36.454954 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.454964 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:36.455071 (+   107us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:36.455100 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2314851584}
I20260812 08:01:36.501902 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.22->[1.23-1.23]
W20260812 08:01:36.503237 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 901ms of latency in SegmentAllocator::Sync()
W20260812 08:01:36.996841 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 41: this message will repeat every 5th retry.
W20260812 08:01:37.270387 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb02c0
W20260812 08:01:37.308266 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 134) took 1037 ms (client timeout 50 ms). Trace:
I20260812 08:01:37.308322 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.308422 31939 rpcz_store.cc:269] 0812 08:01:36.270985 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.271182 (+   197us) service_pool.cc:224] Handling call
0812 08:01:36.271457 (+   275us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:36.271623 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:36.271626 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:36.271805 (+   179us) write_op.cc:270] Start()
0812 08:01:36.271848 (+    43us) write_op.cc:276] Timestamp: P: 1786521696269116 usec, L: 0
0812 08:01:36.272284 (+   436us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:36.272375 (+    91us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:36.272382 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.272397 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:37.308138 (+1035741us) raft_consensus.cc:1685] finished
0812 08:01:37.308141 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.308239 (+    98us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:36.272316 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:36.272443 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:36.272449 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:36.272451 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:36.272804 (+   353us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:36.272817 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:36.272820 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:36.273140 (+   320us) tablet.cc:701] Row locks acquired
0812 08:01:36.273142 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":346,"prepare.run_cpu_time_us":1013,"prepare.run_wall_time_us":1010,"thread_start_us":160,"threads_started":1}]]}
I20260812 08:01:37.308684 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.308856 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 136) took 528 ms (client timeout 50 ms). Trace:
I20260812 08:01:37.308933 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.308921 31942 rpcz_store.cc:269] 0812 08:01:36.780378 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.780474 (+    96us) service_pool.cc:224] Handling call
0812 08:01:37.308263 (+527789us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.308534 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.308541 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.308560 (+    19us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.308563 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.308574 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.308788 (+   214us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:37.308837 (+    49us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1213328256}
W20260812 08:01:37.309011 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 135) took 986 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.309070 31935 rpcz_store.cc:269] 0812 08:01:36.322316 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.322439 (+   123us) service_pool.cc:224] Handling call
0812 08:01:37.308650 (+986211us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.308803 (+   153us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.308807 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.308819 (+    12us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.308822 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.308834 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.308970 (+   136us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:37.308999 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2267413376}
W20260812 08:01:37.309167 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 137) took 477 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.309237 31941 rpcz_store.cc:269] 0812 08:01:36.832071 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.832308 (+   237us) service_pool.cc:224] Handling call
0812 08:01:37.308907 (+476599us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.309007 (+   100us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.309009 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.309016 (+     7us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.309020 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.309030 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.309131 (+   101us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:37.309155 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1095628544}
I20260812 08:01:37.309302 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.23]   Dedup: 1.23->[]
I20260812 08:01:37.309525 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.309544 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 139) took 38 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.309656 31936 rpcz_store.cc:269] 0812 08:01:37.271512 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.271606 (+    94us) service_pool.cc:224] Handling call
0812 08:01:37.309282 (+ 37676us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.309379 (+    97us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.309382 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.309390 (+     8us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.309396 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.309407 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.309516 (+   109us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:37.309535 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86053632}
W20260812 08:01:37.309989 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 138) took 89 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.310058 31938 rpcz_store.cc:269] 0812 08:01:37.220714 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.220816 (+   102us) service_pool.cc:224] Handling call
0812 08:01:37.309492 (+ 88676us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.309697 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.309703 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.309715 (+    12us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.309722 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.309734 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.309939 (+   205us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:37.309973 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203506944}
I20260812 08:01:37.313128 32541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=21, attempt_no=0}) took 22764 ms. Trace:
W20260812 08:01:37.313542 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1152ms of latency in SegmentAllocator::Sync()
I20260812 08:01:37.313305 32541 rpcz_store.cc:276] 0812 08:01:14.548751 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:14.548854 (+   103us) service_pool.cc:224] Handling call
0812 08:01:37.313109 (+22764255us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:35.133253 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:35.133373 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:35.133378 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:35.133381 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:35.133676 (+   295us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:35.133684 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:35.133687 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:36.269104 (+1135417us) lock_manager.cc:598] Waited 1135116us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:36.269121 (+    17us) tablet.cc:701] Row locks acquired
0812 08:01:36.269123 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:36.269174 (+    51us) write_op.cc:270] Start()
0812 08:01:36.269200 (+    26us) write_op.cc:276] Timestamp: P: 1786521696269116 usec, L: 0
0812 08:01:36.269203 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:36.269353 (+   150us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:37.310515 (+1041162us) op_driver.cc:464] REPLICATION: finished
0812 08:01:37.311050 (+   535us) write_op.cc:301] APPLY: starting
0812 08:01:37.311095 (+    45us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:37.311180 (+    85us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:37.311181 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:37.312704 (+  1523us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:37.312769 (+    65us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:37.312775 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:37.312912 (+   137us) log.cc:844] Serialized 319 byte log entry
0812 08:01:37.312971 (+    59us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:37.313021 (+    50us) write_op.cc:454] Released schema lock
0812 08:01:37.313029 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":440,"num_ops":50,"prepare.queue_time_us":20583947,"prepare.run_cpu_time_us":1293,"prepare.run_wall_time_us":1136194,"raft.queue_time_us":1264,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":221,"replication_time_us":1041277,"row_lock_wait_count":1,"row_lock_wait_us":1135116,"thread_start_us":102,"threads_started":1}]]}
W20260812 08:01:37.315104 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1075ms of latency in SegmentAllocator::Sync()
W20260812 08:01:37.404870 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 109) took 903 ms (client timeout 50 ms). Trace:
I20260812 08:01:37.404870 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.405036 32085 rpcz_store.cc:269] 0812 08:01:36.501444 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.501541 (+    97us) service_pool.cc:224] Handling call
0812 08:01:36.501842 (+   301us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:36.502052 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:36.502322 (+   270us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:36.502451 (+   129us) write_op.cc:270] Start()
0812 08:01:36.502477 (+    26us) write_op.cc:276] Timestamp: P: 1786521696269116 usec, L: 0
0812 08:01:36.502831 (+   354us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:36.502874 (+    43us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:36.502880 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:36.502896 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:37.404699 (+901803us) raft_consensus.cc:1685] finished
0812 08:01:37.404702 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.404834 (+   132us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:36.502736 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:36.502945 (+   209us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:36.502949 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:36.502950 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:36.503142 (+   192us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:36.503152 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:36.503154 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:36.504378 (+  1224us) tablet.cc:701] Row locks acquired
0812 08:01:36.504380 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":1713,"row_lock_wait_count":1,"row_lock_wait_us":1018,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}]]}
W20260812 08:01:37.405290 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 110) took 852 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.405344 32088 rpcz_store.cc:269] 0812 08:01:36.552371 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.552504 (+   133us) service_pool.cc:224] Handling call
0812 08:01:37.404806 (+852302us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:37.405083 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.405086 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.405098 (+    12us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.405102 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.405114 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.405256 (+   142us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:37.405279 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959093632}
I20260812 08:01:37.405297 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.23->[]
I20260812 08:01:37.405915 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.405972 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 112) took 459 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.406019 32086 rpcz_store.cc:269] 0812 08:01:36.946646 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.946776 (+   130us) service_pool.cc:224] Handling call
0812 08:01:37.405244 (+458468us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:37.405726 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.405729 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.405737 (+     8us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.405741 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.405753 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.405932 (+   179us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:37.405959 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053434624}
I20260812 08:01:37.406143 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.406173 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 111) took 510 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.406260 32084 rpcz_store.cc:269] 0812 08:01:36.895688 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.895864 (+   176us) service_pool.cc:224] Handling call
0812 08:01:37.405885 (+510021us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:37.406014 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.406017 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.406023 (+     6us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.406025 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.406034 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.406139 (+   105us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:37.406163 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1171644672}
W20260812 08:01:37.406471 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 114) took 67 ms (client timeout 50 ms). Trace:
I20260812 08:01:37.406536 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.23]   Dedup: 1.23->[]
W20260812 08:01:37.406524 32090 rpcz_store.cc:269] 0812 08:01:37.339012 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.339117 (+   105us) service_pool.cc:224] Handling call
0812 08:01:37.406112 (+ 66995us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:37.406243 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.406246 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.406257 (+    11us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.406263 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.406303 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.406432 (+   129us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:37.406457 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153160064}
W20260812 08:01:37.406930 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 113) took 119 ms (client timeout 50 ms). Trace:
W20260812 08:01:37.406996 32089 rpcz_store.cc:269] 0812 08:01:37.287918 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.288023 (+   105us) service_pool.cc:224] Handling call
0812 08:01:37.406507 (+118484us) raft_consensus.cc:1471] Updating replica for 18 ops
0812 08:01:37.406734 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 22
0812 08:01:37.406738 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:37.406747 (+     9us) raft_consensus.cc:1645] Marking committed up to 22
0812 08:01:37.406751 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.406765 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:37.406889 (+   124us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:37.406916 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271435136}
W20260812 08:01:37.415221 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:37.730253 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.24]   Dedup: 1.23->[1.24-1.24]
W20260812 08:01:37.731807 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 958ms of latency in SegmentAllocator::Sync()
W20260812 08:01:38.271826 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 21 similar messages]
W20260812 08:01:38.314493 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0580
W20260812 08:01:38.390695 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 141) took 1076 ms (client timeout 50 ms). Trace:
I20260812 08:01:38.390745 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
I20260812 08:01:38.391078 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.390827 31938 rpcz_store.cc:269] 0812 08:01:37.314066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.314232 (+   166us) service_pool.cc:224] Handling call
0812 08:01:37.314370 (+   138us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:37.314442 (+    72us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:37.314445 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:37.314566 (+   121us) write_op.cc:270] Start()
0812 08:01:37.314620 (+    54us) write_op.cc:276] Timestamp: P: 1786521697313035 usec, L: 0
0812 08:01:37.314879 (+   259us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:37.314934 (+    55us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:37.314940 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.314951 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:38.390550 (+1075599us) raft_consensus.cc:1685] finished
0812 08:01:38.390553 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.390666 (+   113us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:37.314920 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:37.315031 (+   111us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:37.315036 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:37.315038 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:37.315396 (+   358us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:37.315409 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:37.315413 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:37.315711 (+   298us) tablet.cc:701] Row locks acquired
0812 08:01:37.315713 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":866,"prepare.run_wall_time_us":866,"thread_start_us":64,"threads_started":1}]]}
W20260812 08:01:38.391228 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 143) took 575 ms (client timeout 50 ms). Trace:
I20260812 08:01:38.391312 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.391350 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 145) took 169 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.391400 31942 rpcz_store.cc:269] 0812 08:01:38.221781 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.221917 (+   136us) service_pool.cc:224] Handling call
0812 08:01:38.391055 (+169138us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.391191 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.391193 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.391199 (+     6us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.391202 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.391211 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.391316 (+   105us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:38.391339 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388558208}
W20260812 08:01:38.391291 31941 rpcz_store.cc:269] 0812 08:01:37.815800 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.815984 (+   184us) service_pool.cc:224] Handling call
0812 08:01:38.390706 (+574722us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.391000 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.391003 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.391014 (+    11us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.391018 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.391029 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.391182 (+   153us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:38.391215 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320460160}
W20260812 08:01:38.391608 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 142) took 1026 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.391661 31936 rpcz_store.cc:269] 0812 08:01:37.365080 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.365191 (+   111us) service_pool.cc:224] Handling call
0812 08:01:38.391277 (+1026086us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.391429 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.391433 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.391441 (+     8us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.391444 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.391456 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.391568 (+   112us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:38.391596 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2359317760}
I20260812 08:01:38.391745 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
I20260812 08:01:38.392017 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.392052 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 146) took 119 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.392108 31939 rpcz_store.cc:269] 0812 08:01:38.272906 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.272987 (+    81us) service_pool.cc:224] Handling call
0812 08:01:38.391722 (+118735us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.391842 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.391846 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.391854 (+     8us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.391860 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.391870 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.392006 (+   136us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:38.392031 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272475904}
W20260812 08:01:38.392336 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 144) took 524 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.392395 31935 rpcz_store.cc:269] 0812 08:01:37.867617 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.867766 (+   149us) service_pool.cc:224] Handling call
0812 08:01:38.391985 (+524219us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.392125 (+   140us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.392131 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.392138 (+     7us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.392141 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.392151 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.392267 (+   116us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:38.392297 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205124352}
I20260812 08:01:38.664235 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.668486 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 972ms of latency in SegmentAllocator::Sync()
I20260812 08:01:38.668728 32549 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=22, attempt_no=0}) took 22926 ms. Trace:
I20260812 08:01:38.668836 32549 rpcz_store.cc:276] 0812 08:01:15.741908 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:15.742024 (+   116us) service_pool.cc:224] Handling call
0812 08:01:38.668714 (+22926690us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:36.269480 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:36.269601 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:36.269605 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:36.269606 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:36.269901 (+   295us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:36.269910 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:36.269915 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:37.313032 (+1043117us) lock_manager.cc:598] Waited 1042815us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:37.313045 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:37.313046 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:37.313125 (+    79us) write_op.cc:270] Start()
0812 08:01:37.313161 (+    36us) write_op.cc:276] Timestamp: P: 1786521697313035 usec, L: 0
0812 08:01:37.313165 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:37.313356 (+   191us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:38.665589 (+1352233us) op_driver.cc:464] REPLICATION: finished
0812 08:01:38.666169 (+   580us) write_op.cc:301] APPLY: starting
0812 08:01:38.666205 (+    36us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:38.666299 (+    94us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:38.666301 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:38.667598 (+  1297us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:38.667650 (+    52us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:38.667656 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:38.667871 (+   215us) log.cc:844] Serialized 319 byte log entry
0812 08:01:38.667987 (+   116us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:38.668626 (+   639us) write_op.cc:454] Released schema lock
0812 08:01:38.668636 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":485,"num_ops":50,"prepare.queue_time_us":20527019,"prepare.run_cpu_time_us":1443,"prepare.run_wall_time_us":1043985,"raft.queue_time_us":161,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"replication_time_us":1352386,"row_lock_wait_count":1,"row_lock_wait_us":1042815,"spinlock_wait_cycles":77056,"thread_start_us":310,"threads_started":1}]]}
W20260812 08:01:38.669873 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1014ms of latency in SegmentAllocator::Sync()
W20260812 08:01:38.690356 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 115) took 960 ms (client timeout 50 ms). Trace:
I20260812 08:01:38.690410 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.690486 32089 rpcz_store.cc:269] 0812 08:01:37.729522 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.729653 (+   131us) service_pool.cc:224] Handling call
0812 08:01:37.730178 (+   525us) raft_consensus.cc:1471] Updating replica for 19 ops
0812 08:01:37.730411 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:37.730861 (+   450us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:37.731010 (+   149us) write_op.cc:270] Start()
0812 08:01:37.731070 (+    60us) write_op.cc:276] Timestamp: P: 1786521697313035 usec, L: 0
0812 08:01:37.731478 (+   408us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:37.731531 (+    53us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:37.731538 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:37.731555 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:38.690232 (+958677us) raft_consensus.cc:1685] finished
0812 08:01:38.690235 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.690327 (+    92us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:37.731477 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:37.731622 (+   145us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:37.731627 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:37.731629 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:37.731838 (+   209us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:37.731847 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:37.731850 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:37.732991 (+  1141us) tablet.cc:701] Row locks acquired
0812 08:01:37.732993 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":317,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":1581,"row_lock_wait_count":1,"row_lock_wait_us":839,"thread_start_us":122,"threads_started":1}]]}
I20260812 08:01:38.690755 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.690817 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 116) took 910 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.690871 32090 rpcz_store.cc:269] 0812 08:01:37.780684 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.780827 (+   143us) service_pool.cc:224] Handling call
0812 08:01:38.690357 (+909530us) raft_consensus.cc:1471] Updating replica for 19 ops
0812 08:01:38.690573 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.690577 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.690590 (+    13us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.690622 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.690641 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.690765 (+   124us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:38.690804 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2090759424}
I20260812 08:01:38.691056 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.24]   Dedup: 1.24->[]
W20260812 08:01:38.691079 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 118) took 336 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.691217 32086 rpcz_store.cc:269] 0812 08:01:38.354444 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.354645 (+   201us) service_pool.cc:224] Handling call
0812 08:01:38.690718 (+336073us) raft_consensus.cc:1471] Updating replica for 19 ops
0812 08:01:38.690899 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.690902 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.690909 (+     7us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.690913 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.690924 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.691039 (+   115us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:38.691066 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":771984640}
W20260812 08:01:38.691407 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 117) took 388 ms (client timeout 50 ms). Trace:
W20260812 08:01:38.691471 32084 rpcz_store.cc:269] 0812 08:01:38.303341 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.303453 (+   112us) service_pool.cc:224] Handling call
0812 08:01:38.691016 (+387563us) raft_consensus.cc:1471] Updating replica for 19 ops
0812 08:01:38.691236 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 23
0812 08:01:38.691239 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:38.691246 (+     7us) raft_consensus.cc:1645] Marking committed up to 23
0812 08:01:38.691249 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.691259 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:38.691367 (+   108us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:38.691396 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890485504}
I20260812 08:01:38.705783 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.25]   Dedup: 1.24->[1.25-1.25]
W20260812 08:01:38.707300 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 762ms of latency in SegmentAllocator::Sync()
W20260812 08:01:38.770766 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:39.233675 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 46: this message will repeat every 5th retry.
W20260812 08:01:39.469944 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 119) took 765 ms (client timeout 50 ms). Trace:
I20260812 08:01:39.469944 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.470115 32084 rpcz_store.cc:269] 0812 08:01:38.704933 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.705255 (+   322us) service_pool.cc:224] Handling call
0812 08:01:38.705676 (+   421us) raft_consensus.cc:1471] Updating replica for 20 ops
0812 08:01:38.705940 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:38.706233 (+   293us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:38.706426 (+   193us) write_op.cc:270] Start()
0812 08:01:38.706456 (+    30us) write_op.cc:276] Timestamp: P: 1786521698668113 usec, L: 0
0812 08:01:38.706958 (+   502us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:38.707020 (+    62us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:38.707028 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.707045 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:39.469768 (+762723us) raft_consensus.cc:1685] finished
0812 08:01:39.469771 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.469909 (+   138us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:38.706734 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:38.706968 (+   234us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:38.706975 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:38.706979 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:38.707312 (+   333us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:38.707323 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:38.707326 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:38.708293 (+   967us) tablet.cc:701] Row locks acquired
0812 08:01:38.708295 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":1236,"prepare.run_wall_time_us":1786,"row_lock_wait_count":1,"row_lock_wait_us":623,"spinlock_wait_cycles":318208,"thread_start_us":114,"threads_started":1}]]}
W20260812 08:01:39.470424 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 121) took 337 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.470475 32090 rpcz_store.cc:269] 0812 08:01:39.132687 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.132802 (+   115us) service_pool.cc:224] Handling call
0812 08:01:39.469887 (+337085us) raft_consensus.cc:1471] Updating replica for 20 ops
0812 08:01:39.470210 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.470213 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.470225 (+    12us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.470229 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.470241 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.470382 (+   141us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:39.470411 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774334208}
I20260812 08:01:39.470698 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.25]   Dedup: 1.25->[]
I20260812 08:01:39.471056 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.471138 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 122) took 287 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.471189 32089 rpcz_store.cc:269] 0812 08:01:39.183562 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.183742 (+   180us) service_pool.cc:224] Handling call
0812 08:01:39.470604 (+286862us) raft_consensus.cc:1471] Updating replica for 20 ops
0812 08:01:39.470909 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.470911 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.470920 (+     9us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.470923 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.470932 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.471089 (+   157us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:39.471123 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":657786624}
W20260812 08:01:39.471342 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 120) took 715 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.471407 32086 rpcz_store.cc:269] 0812 08:01:38.755834 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.755985 (+   151us) service_pool.cc:224] Handling call
0812 08:01:39.471013 (+715028us) raft_consensus.cc:1471] Updating replica for 20 ops
0812 08:01:39.471175 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.471178 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.471187 (+     9us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.471190 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.471203 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.471310 (+   107us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:39.471333 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1643504384}
I20260812 08:01:39.667421 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.669356 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0840
I20260812 08:01:39.671034 32557 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=23, attempt_no=0}) took 22820 ms. Trace:
I20260812 08:01:39.671159 32557 rpcz_store.cc:276] 0812 08:01:16.850929 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:16.851134 (+   205us) service_pool.cc:224] Handling call
0812 08:01:39.671011 (+22819877us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:37.313504 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:37.313629 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:37.313633 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:37.313635 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:37.314018 (+   383us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:37.314027 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:37.314033 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:38.668022 (+1353989us) lock_manager.cc:598] Waited 1353669us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:38.668035 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:38.668037 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:38.668093 (+    56us) write_op.cc:270] Start()
0812 08:01:38.668116 (+    23us) write_op.cc:276] Timestamp: P: 1786521698668113 usec, L: 0
0812 08:01:38.668117 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:01:38.668270 (+   153us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:39.668555 (+1000285us) op_driver.cc:464] REPLICATION: finished
0812 08:01:39.668856 (+   301us) write_op.cc:301] APPLY: starting
0812 08:01:39.668890 (+    34us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:39.668954 (+    64us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:39.668956 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:39.670569 (+  1613us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:39.670649 (+    80us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:39.670661 (+    12us) write_op.cc:312] APPLY: finished
0812 08:01:39.670805 (+   144us) log.cc:844] Serialized 319 byte log entry
0812 08:01:39.670874 (+    69us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:39.670906 (+    32us) write_op.cc:454] Released schema lock
0812 08:01:39.670917 (+    11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":190,"num_ops":50,"prepare.queue_time_us":20461916,"prepare.run_cpu_time_us":1489,"prepare.run_wall_time_us":1354882,"raft.queue_time_us":115,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":203,"replication_time_us":1000399,"row_lock_wait_count":1,"row_lock_wait_us":1353669,"thread_start_us":91,"threads_started":1}]]}
W20260812 08:01:39.671547 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1061ms of latency in SegmentAllocator::Sync()
W20260812 08:01:39.674362 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1111ms of latency in SegmentAllocator::Sync()
I20260812 08:01:39.684579 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.684742 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 149) took 1015 ms (client timeout 50 ms). Trace:
I20260812 08:01:39.685057 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.685181 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 152) took 589 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.685264 31941 rpcz_store.cc:269] 0812 08:01:39.096180 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.096285 (+   105us) service_pool.cc:224] Handling call
0812 08:01:39.684525 (+588240us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.684940 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.684943 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.684954 (+    11us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.684956 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.684967 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.685129 (+   162us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:39.685167 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1351407872}
I20260812 08:01:39.685398 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.685484 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 150) took 964 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.685046 31935 rpcz_store.cc:269] 0812 08:01:38.668915 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.668968 (+    53us) service_pool.cc:224] Handling call
0812 08:01:38.669109 (+   141us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:38.669193 (+    84us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:38.669196 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:38.669327 (+   131us) write_op.cc:270] Start()
0812 08:01:38.669355 (+    28us) write_op.cc:276] Timestamp: P: 1786521698668113 usec, L: 0
0812 08:01:38.669688 (+   333us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:38.670004 (+   316us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:38.670011 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:38.670024 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:39.684446 (+1014422us) raft_consensus.cc:1685] finished
0812 08:01:39.684449 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.684715 (+   266us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:38.669897 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:38.670033 (+   136us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:38.670038 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:38.670041 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:38.670400 (+   359us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:38.670429 (+    29us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:38.670433 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:38.670920 (+   487us) tablet.cc:701] Row locks acquired
0812 08:01:38.670922 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":437,"prepare.run_cpu_time_us":1111,"prepare.run_wall_time_us":1110,"thread_start_us":105,"threads_started":1}]]}
W20260812 08:01:39.685532 31939 rpcz_store.cc:269] 0812 08:01:38.720595 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.720715 (+   120us) service_pool.cc:224] Handling call
0812 08:01:39.685030 (+964315us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.685292 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.685296 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.685306 (+    10us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.685312 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.685327 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.685446 (+   119us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:39.685473 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2217246080}
W20260812 08:01:39.685657 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 154) took 120 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.685714 31938 rpcz_store.cc:269] 0812 08:01:39.565035 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.565119 (+    84us) service_pool.cc:224] Handling call
0812 08:01:39.685375 (+120256us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.685490 (+   115us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.685492 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.685500 (+     8us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.685503 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.685511 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.685617 (+   106us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:39.685645 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276118272}
I20260812 08:01:39.685834 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25]   Dedup: 1.25->[]
I20260812 08:01:39.685969 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25]   Dedup: 1.25->[]
W20260812 08:01:39.686054 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 153) took 172 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.686111 31942 rpcz_store.cc:269] 0812 08:01:39.513970 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.514191 (+   221us) service_pool.cc:224] Handling call
0812 08:01:39.685812 (+171621us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.685910 (+    98us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.685913 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.685919 (+     6us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.685922 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.685930 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.686021 (+    91us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:39.686045 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394266880}
W20260812 08:01:39.686198 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 151) took 641 ms (client timeout 50 ms). Trace:
W20260812 08:01:39.686235 31936 rpcz_store.cc:269] 0812 08:01:39.045002 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.045228 (+   226us) service_pool.cc:224] Handling call
0812 08:01:39.685948 (+640720us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.686052 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 24
0812 08:01:39.686054 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:39.686060 (+     6us) raft_consensus.cc:1645] Marking committed up to 24
0812 08:01:39.686067 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.686077 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:39.686163 (+    86us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:39.686180 (+    17us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1472899456}
W20260812 08:01:39.773633 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:40.056502 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.25->[1.26-1.26]
W20260812 08:01:40.057847 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1001ms of latency in SegmentAllocator::Sync()
W20260812 08:01:40.672274 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0b00
I20260812 08:01:40.786207 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26]   Dedup: 1.26->[]
W20260812 08:01:40.786249 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 125) took 1113 ms (client timeout 50 ms). Trace:
I20260812 08:01:40.786903 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26]   Dedup: 1.26->[]
W20260812 08:01:40.786760 32089 rpcz_store.cc:269] 0812 08:01:39.672873 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.672945 (+    72us) service_pool.cc:224] Handling call
0812 08:01:39.673112 (+   167us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:39.673179 (+    67us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:39.673182 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:39.673443 (+   261us) write_op.cc:270] Start()
0812 08:01:39.673469 (+    26us) write_op.cc:276] Timestamp: P: 1786521699671063 usec, L: 0
0812 08:01:39.674043 (+   574us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:39.674088 (+    45us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:39.674095 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:39.674109 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:40.785980 (+1111871us) raft_consensus.cc:1685] finished
0812 08:01:40.785984 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.786139 (+   155us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:39.673953 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:39.674073 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:39.674078 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:39.674079 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:39.674658 (+   579us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:39.674671 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:39.674682 (+    11us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:39.674863 (+   181us) tablet.cc:701] Row locks acquired
0812 08:01:39.674864 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":355,"prepare.run_cpu_time_us":982,"prepare.run_wall_time_us":981,"thread_start_us":198,"threads_started":1}]]}
W20260812 08:01:40.787075 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 127) took 658 ms (client timeout 50 ms). Trace:
I20260812 08:01:40.787184 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26]   Dedup: 1.26->[]
W20260812 08:01:40.787200 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 126) took 1063 ms (client timeout 50 ms). Trace:
I20260812 08:01:40.787441 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26]   Dedup: 1.26->[]
W20260812 08:01:40.787361 32086 rpcz_store.cc:269] 0812 08:01:39.723592 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.723712 (+   120us) service_pool.cc:224] Handling call
0812 08:01:40.786876 (+1063164us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:40.787042 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.787044 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:40.787050 (+     6us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.787054 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.787063 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.787167 (+   104us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:40.787189 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2444580736}
W20260812 08:01:40.787513 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 130) took 181 ms (client timeout 50 ms). Trace:
W20260812 08:01:40.787583 32085 rpcz_store.cc:269] 0812 08:01:40.605963 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.606037 (+    74us) service_pool.cc:224] Handling call
0812 08:01:40.787144 (+181107us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:40.787324 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.787327 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:40.787335 (+     8us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.787342 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.787356 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.787472 (+   116us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:40.787502 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":415564544}
I20260812 08:01:40.787727 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26]   Dedup: 1.26->[]
W20260812 08:01:40.787236 32090 rpcz_store.cc:269] 0812 08:01:40.129110 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.129226 (+   116us) service_pool.cc:224] Handling call
0812 08:01:40.786147 (+656921us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:40.786810 (+   663us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.786814 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:40.786828 (+    14us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.786831 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.786846 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.787003 (+   157us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:40.787062 (+    59us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1510232448}
W20260812 08:01:40.787704 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 129) took 232 ms (client timeout 50 ms). Trace:
W20260812 08:01:40.787849 32088 rpcz_store.cc:269] 0812 08:01:40.554836 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.554976 (+   140us) service_pool.cc:224] Handling call
0812 08:01:40.787416 (+232440us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:40.787534 (+   118us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.787537 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:40.787544 (+     7us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.787547 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.787556 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.787662 (+   106us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:40.787691 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":534065024}
W20260812 08:01:40.787982 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 128) took 607 ms (client timeout 50 ms). Trace:
W20260812 08:01:40.788033 32084 rpcz_store.cc:269] 0812 08:01:40.180161 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.180293 (+   132us) service_pool.cc:224] Handling call
0812 08:01:40.787698 (+607405us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:40.787823 (+   125us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.787825 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:40.787832 (+     7us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.787835 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.787843 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:40.787944 (+   101us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:40.787970 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1396434688}
W20260812 08:01:41.059480 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 155) took 1003 ms (client timeout 50 ms). Trace:
I20260812 08:01:41.059640 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.26->[]
W20260812 08:01:41.059700 31936 rpcz_store.cc:269] 0812 08:01:40.055934 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.056081 (+   147us) service_pool.cc:224] Handling call
0812 08:01:40.056415 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:40.056667 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:40.056932 (+   265us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:40.057108 (+   176us) write_op.cc:270] Start()
0812 08:01:40.057135 (+    27us) write_op.cc:276] Timestamp: P: 1786521699671063 usec, L: 0
0812 08:01:40.057469 (+   334us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:40.057514 (+    45us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:40.057520 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:40.057535 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:41.059360 (+1001825us) raft_consensus.cc:1685] finished
0812 08:01:41.059362 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.059454 (+    92us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:40.057442 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:40.057599 (+   157us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:40.057604 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:40.057606 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:40.058236 (+   630us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:40.058247 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:40.058251 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:40.058893 (+   642us) tablet.cc:701] Row locks acquired
0812 08:01:40.058895 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":207,"prepare.run_cpu_time_us":1155,"prepare.run_wall_time_us":1535,"row_lock_wait_count":1,"row_lock_wait_us":445,"thread_start_us":69,"threads_started":1}]]}
I20260812 08:01:41.060041 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.26->[]
W20260812 08:01:41.060122 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 156) took 953 ms (client timeout 50 ms). Trace:
W20260812 08:01:41.060773 31942 rpcz_store.cc:269] 0812 08:01:40.107185 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.107320 (+   135us) service_pool.cc:224] Handling call
0812 08:01:41.059585 (+952265us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:41.059902 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:41.059905 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.059916 (+    11us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:41.059919 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.059930 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.060068 (+   138us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.060106 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2189296384}
I20260812 08:01:41.060961 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.26->[]
W20260812 08:01:41.061017 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 158) took 479 ms (client timeout 50 ms). Trace:
W20260812 08:01:41.061088 31935 rpcz_store.cc:269] 0812 08:01:40.581993 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.582121 (+   128us) service_pool.cc:224] Handling call
0812 08:01:41.060006 (+477885us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:41.060814 (+   808us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:41.060818 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.060829 (+    11us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:41.060834 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.060847 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.060975 (+   128us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.061004 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1098452864}
W20260812 08:01:41.061306 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 157) took 530 ms (client timeout 50 ms). Trace:
I20260812 08:01:41.061353 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.26->[]
W20260812 08:01:41.061371 31938 rpcz_store.cc:269] 0812 08:01:40.530718 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.530856 (+   138us) service_pool.cc:224] Handling call
0812 08:01:41.060927 (+530071us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:41.061092 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:41.061098 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.061110 (+    12us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:41.061114 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.061129 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.061259 (+   130us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.061291 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1218410368}
I20260812 08:01:41.061553 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.24->[1.25-1.26]   Dedup: 1.26->[]
W20260812 08:01:41.061707 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 159) took 87 ms (client timeout 50 ms). Trace:
W20260812 08:01:41.061807 31939 rpcz_store.cc:269] 0812 08:01:40.974490 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:40.974640 (+   150us) service_pool.cc:224] Handling call
0812 08:01:41.061323 (+ 86683us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:41.061443 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 25
0812 08:01:41.061446 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.061453 (+     7us) raft_consensus.cc:1645] Marking committed up to 25
0812 08:01:41.061457 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.061467 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.061582 (+   115us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.061687 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":198933248}
I20260812 08:01:41.065814 32564 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=24, attempt_no=0}) took 23063 ms. Trace:
I20260812 08:01:41.065937 32564 rpcz_store.cc:276] 0812 08:01:18.001957 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.002037 (+    80us) service_pool.cc:224] Handling call
0812 08:01:41.065792 (+23063755us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:38.668420 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:38.668538 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:38.668543 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:38.668544 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:38.668841 (+   297us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:38.668850 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:38.668854 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:39.671033 (+1002179us) lock_manager.cc:598] Waited 1001784us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:39.671055 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:39.671057 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:39.671098 (+    41us) write_op.cc:270] Start()
0812 08:01:39.671121 (+    23us) write_op.cc:276] Timestamp: P: 1786521699671063 usec, L: 0
0812 08:01:39.671122 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:01:39.671310 (+   188us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:41.062846 (+1391536us) op_driver.cc:464] REPLICATION: finished
0812 08:01:41.063363 (+   517us) write_op.cc:301] APPLY: starting
0812 08:01:41.063402 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:41.063471 (+    69us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:41.063473 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:41.065206 (+  1733us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:41.065265 (+    59us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:41.065271 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:41.065435 (+   164us) log.cc:844] Serialized 319 byte log entry
0812 08:01:41.065655 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:41.065706 (+    51us) write_op.cc:454] Released schema lock
0812 08:01:41.065715 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":442,"num_ops":50,"prepare.queue_time_us":20666001,"prepare.run_cpu_time_us":1439,"prepare.run_wall_time_us":1003026,"raft.queue_time_us":1704,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":158,"replication_time_us":1391490,"row_lock_wait_count":1,"row_lock_wait_us":1001784,"thread_start_us":146,"threads_started":1}]]}
W20260812 08:01:41.066511 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1018ms of latency in SegmentAllocator::Sync()
W20260812 08:01:41.068586 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 871ms of latency in SegmentAllocator::Sync()
W20260812 08:01:41.169077 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:41.169297 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.26->[1.27-1.27]
W20260812 08:01:41.170468 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1036ms of latency in SegmentAllocator::Sync()
I20260812 08:01:41.940481 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27]   Dedup: 1.27->[]
W20260812 08:01:41.940500 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 162) took 873 ms (client timeout 50 ms). Trace:
I20260812 08:01:41.941079 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27]   Dedup: 1.27->[]
W20260812 08:01:41.940904 31939 rpcz_store.cc:269] 0812 08:01:41.067257 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.067315 (+    58us) service_pool.cc:224] Handling call
0812 08:01:41.067485 (+   170us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:41.067560 (+    75us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:41.067563 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:41.067678 (+   115us) write_op.cc:270] Start()
0812 08:01:41.067703 (+    25us) write_op.cc:276] Timestamp: P: 1786521701065774 usec, L: 0
0812 08:01:41.068286 (+   583us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:41.068334 (+    48us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:41.068341 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.068355 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:41.940142 (+871787us) raft_consensus.cc:1685] finished
0812 08:01:41.940145 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.940248 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:41.068263 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:41.068388 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:41.068392 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:41.068394 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:41.068726 (+   332us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:41.068739 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:41.068742 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:41.068966 (+   224us) tablet.cc:701] Row locks acquired
0812 08:01:41.068967 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":464,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":775,"thread_start_us":267,"threads_started":1}]]}
W20260812 08:01:41.941205 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 165) took 285 ms (client timeout 50 ms). Trace:
W20260812 08:01:41.941255 31935 rpcz_store.cc:269] 0812 08:01:41.655962 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.656077 (+   115us) service_pool.cc:224] Handling call
0812 08:01:41.940273 (+284196us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:41.940716 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:41.940722 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.940933 (+   211us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:41.940946 (+    13us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.940971 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.941144 (+   173us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.941181 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":653096320}
W20260812 08:01:41.941478 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 164) took 336 ms (client timeout 50 ms). Trace:
I20260812 08:01:41.941507 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27]   Dedup: 1.27->[]
W20260812 08:01:41.941529 31938 rpcz_store.cc:269] 0812 08:01:41.604610 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.604760 (+   150us) service_pool.cc:224] Handling call
0812 08:01:41.941046 (+336286us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:41.941300 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:41.941303 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.941312 (+     9us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:41.941316 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.941327 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.941435 (+   108us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.941465 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":772868480}
W20260812 08:01:41.941946 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 163) took 823 ms (client timeout 50 ms). Trace:
W20260812 08:01:41.942035 31941 rpcz_store.cc:269] 0812 08:01:41.118730 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.118855 (+   125us) service_pool.cc:224] Handling call
0812 08:01:41.941482 (+822627us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:41.941586 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:41.941588 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:41.941595 (+     7us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:41.941598 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.941607 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:41.941703 (+    96us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:41.941926 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1891394816}
I20260812 08:01:42.033499 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.067250 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0dc0
W20260812 08:01:42.087960 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1085ms of latency in SegmentAllocator::Sync()
I20260812 08:01:42.088400 32573 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=25, attempt_no=0}) took 23200 ms. Trace:
I20260812 08:01:42.088480 32573 rpcz_store.cc:276] 0812 08:01:18.887938 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:18.888051 (+   113us) service_pool.cc:224] Handling call
0812 08:01:42.088294 (+23200243us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:39.671504 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:39.671656 (+   152us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:39.671663 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:39.671667 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:39.671902 (+   235us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:39.671909 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:39.671912 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:41.065723 (+1393811us) lock_manager.cc:598] Waited 1393617us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:41.065735 (+    12us) tablet.cc:701] Row locks acquired
0812 08:01:41.065737 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:41.065810 (+    73us) write_op.cc:270] Start()
0812 08:01:41.065836 (+    26us) write_op.cc:276] Timestamp: P: 1786521701065774 usec, L: 0
0812 08:01:41.065840 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:41.066003 (+   163us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:42.085213 (+1019210us) op_driver.cc:464] REPLICATION: finished
0812 08:01:42.085548 (+   335us) write_op.cc:301] APPLY: starting
0812 08:01:42.085597 (+    49us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:42.085684 (+    87us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:42.085686 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:42.087178 (+  1492us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:42.087232 (+    54us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:42.087239 (+     7us) write_op.cc:312] APPLY: finished
0812 08:01:42.087374 (+   135us) log.cc:844] Serialized 319 byte log entry
0812 08:01:42.087458 (+    84us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:42.088189 (+   731us) write_op.cc:454] Released schema lock
0812 08:01:42.088213 (+    24us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":255,"num_ops":50,"prepare.queue_time_us":20783016,"prepare.run_cpu_time_us":1353,"prepare.run_wall_time_us":1394674,"raft.queue_time_us":365,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":406,"replication_time_us":1019311,"row_lock_wait_count":1,"row_lock_wait_us":1393617,"spinlock_wait_cycles":52992,"thread_start_us":115,"threads_started":1}]]}
W20260812 08:01:42.089643 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1043ms of latency in SegmentAllocator::Sync()
W20260812 08:01:42.156237 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:01:42.190238 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:42.207129 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 131) took 1038 ms (client timeout 50 ms). Trace:
I20260812 08:01:42.207129 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.207392 32084 rpcz_store.cc:269] 0812 08:01:41.168886 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.169009 (+   123us) service_pool.cc:224] Handling call
0812 08:01:41.169208 (+   199us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:41.169451 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:41.169674 (+   223us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:41.169763 (+    89us) write_op.cc:270] Start()
0812 08:01:41.169794 (+    31us) write_op.cc:276] Timestamp: P: 1786521701065774 usec, L: 0
0812 08:01:41.170138 (+   344us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:41.170184 (+    46us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:41.170190 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:41.170205 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:42.206953 (+1036748us) raft_consensus.cc:1685] finished
0812 08:01:42.206956 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.207093 (+   137us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:41.170034 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:41.170158 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:41.170163 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:41.170165 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:41.170391 (+   226us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:41.170401 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:41.170403 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:41.171692 (+  1289us) tablet.cc:701] Row locks acquired
0812 08:01:41.171694 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":1053,"prepare.run_wall_time_us":1978,"row_lock_wait_count":1,"row_lock_wait_us":1027,"thread_start_us":90,"threads_started":1}]]}
I20260812 08:01:42.207626 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.207698 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 132) took 987 ms (client timeout 50 ms). Trace:
W20260812 08:01:42.207762 32085 rpcz_store.cc:269] 0812 08:01:41.219977 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.220059 (+    82us) service_pool.cc:224] Handling call
0812 08:01:42.207073 (+987014us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:42.207463 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:42.207467 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:42.207480 (+    13us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:42.207485 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.207498 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.207651 (+   153us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:42.207683 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2269453696}
I20260812 08:01:42.207922 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.208016 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 134) took 574 ms (client timeout 50 ms). Trace:
W20260812 08:01:42.208065 32090 rpcz_store.cc:269] 0812 08:01:41.633404 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.633761 (+   357us) service_pool.cc:224] Handling call
0812 08:01:42.207591 (+573830us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:42.207801 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:42.207807 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:42.207817 (+    10us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:42.207824 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.207840 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.207968 (+   128us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:42.208003 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1318860672}
W20260812 08:01:42.208169 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 133) took 625 ms (client timeout 50 ms). Trace:
I20260812 08:01:42.208166 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.208246 32088 rpcz_store.cc:269] 0812 08:01:41.582733 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.582831 (+    98us) service_pool.cc:224] Handling call
0812 08:01:42.207893 (+625062us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:42.208012 (+   119us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:42.208015 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:42.208022 (+     7us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:42.208029 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.208041 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.208140 (+    99us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:42.208160 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1436978048}
I20260812 08:01:42.208487 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.27]   Dedup: 1.27->[]
W20260812 08:01:42.208575 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 136) took 102 ms (client timeout 50 ms). Trace:
W20260812 08:01:42.208635 32089 rpcz_store.cc:269] 0812 08:01:42.106306 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.106438 (+   132us) service_pool.cc:224] Handling call
0812 08:01:42.208140 (+101702us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:42.208375 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:42.208379 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:42.208388 (+     9us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:42.208396 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.208410 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.208533 (+   123us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:42.208562 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233392896}
W20260812 08:01:42.208736 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 135) took 154 ms (client timeout 50 ms). Trace:
W20260812 08:01:42.208963 32086 rpcz_store.cc:269] 0812 08:01:42.054501 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.054643 (+   142us) service_pool.cc:224] Handling call
0812 08:01:42.208457 (+153814us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:42.208579 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 26
0812 08:01:42.208585 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:42.208594 (+     9us) raft_consensus.cc:1645] Marking committed up to 26
0812 08:01:42.208598 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.208608 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:42.208707 (+    99us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:42.208727 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":353245568}
I20260812 08:01:42.665307 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.28]   Dedup: 1.27->[1.28-1.28]
W20260812 08:01:42.666489 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 824ms of latency in SegmentAllocator::Sync()
W20260812 08:01:43.088819 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1080
W20260812 08:01:43.133455 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 168) took 1044 ms (client timeout 50 ms). Trace:
I20260812 08:01:43.133582 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.133598 31941 rpcz_store.cc:269] 0812 08:01:42.088603 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.088685 (+    82us) service_pool.cc:224] Handling call
0812 08:01:42.088839 (+   154us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:42.088929 (+    90us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:42.088936 (+     7us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:42.089078 (+   142us) write_op.cc:270] Start()
0812 08:01:42.089103 (+    25us) write_op.cc:276] Timestamp: P: 1786521702087503 usec, L: 0
0812 08:01:42.089412 (+   309us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:42.089445 (+    33us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:42.089451 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.089464 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:43.133207 (+1043743us) raft_consensus.cc:1685] finished
0812 08:01:43.133210 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.133423 (+   213us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:42.089412 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:42.089524 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:42.089527 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:42.089529 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:42.089913 (+   384us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:42.089927 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:42.089930 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:42.090227 (+   297us) tablet.cc:701] Row locks acquired
0812 08:01:42.090229 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":221,"prepare.run_cpu_time_us":874,"prepare.run_wall_time_us":874,"thread_start_us":83,"threads_started":1}]]}
I20260812 08:01:43.134094 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.134250 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 169) took 994 ms (client timeout 50 ms). Trace:
I20260812 08:01:43.134316 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.134405 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 171) took 522 ms (client timeout 50 ms). Trace:
W20260812 08:01:43.134325 31938 rpcz_store.cc:269] 0812 08:01:42.140085 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.140210 (+   125us) service_pool.cc:224] Handling call
0812 08:01:43.133535 (+993325us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.133953 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.133957 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.133969 (+    12us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.133975 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.133993 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.134189 (+   196us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:43.134231 (+    42us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2283923840}
W20260812 08:01:43.134465 31939 rpcz_store.cc:269] 0812 08:01:42.612281 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.612573 (+   292us) service_pool.cc:224] Handling call
0812 08:01:43.134065 (+521492us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.134204 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.134210 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.134221 (+    11us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.134228 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.134241 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.134366 (+   125us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:43.134394 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1198773632}
I20260812 08:01:43.134620 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.134657 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 172) took 173 ms (client timeout 50 ms). Trace:
W20260812 08:01:43.134712 31942 rpcz_store.cc:269] 0812 08:01:42.961107 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.961238 (+   131us) service_pool.cc:224] Handling call
0812 08:01:43.134290 (+173052us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.134438 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.134443 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.134453 (+    10us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.134457 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.134468 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.134609 (+   141us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:43.134634 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397550464}
W20260812 08:01:43.134912 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 170) took 574 ms (client timeout 50 ms). Trace:
I20260812 08:01:43.134980 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.134975 31935 rpcz_store.cc:269] 0812 08:01:42.560791 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.561038 (+   247us) service_pool.cc:224] Handling call
0812 08:01:43.134557 (+573519us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.134720 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.134723 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.134733 (+    10us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.134740 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.134752 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.134869 (+   117us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:43.134898 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1318225792}
W20260812 08:01:43.135372 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 173) took 123 ms (client timeout 50 ms). Trace:
W20260812 08:01:43.135442 31936 rpcz_store.cc:269] 0812 08:01:43.012286 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.012424 (+   138us) service_pool.cc:224] Handling call
0812 08:01:43.134956 (+122532us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.135111 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.135114 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.135122 (+     8us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.135129 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.135141 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.135329 (+   188us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:43.135359 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281351424}
I20260812 08:01:43.371585 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28]   Dedup: 1.28->[]
I20260812 08:01:43.375332 32580 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=26, attempt_no=0}) took 23462 ms. Trace:
W20260812 08:01:43.375726 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1038ms of latency in SegmentAllocator::Sync()
I20260812 08:01:43.375471 32580 rpcz_store.cc:276] 0812 08:01:19.913100 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:19.913183 (+    83us) service_pool.cc:224] Handling call
0812 08:01:43.375310 (+23462127us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:41.066208 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:41.066371 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:41.066377 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:41.066381 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:41.066770 (+   389us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:41.066780 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:41.066784 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:42.087496 (+1020712us) lock_manager.cc:598] Waited 1020423us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:42.087510 (+    14us) tablet.cc:701] Row locks acquired
0812 08:01:42.087511 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:42.087561 (+    50us) write_op.cc:270] Start()
0812 08:01:42.087583 (+    22us) write_op.cc:276] Timestamp: P: 1786521702087503 usec, L: 0
0812 08:01:42.087585 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:42.087733 (+   148us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:43.372868 (+1285135us) op_driver.cc:464] REPLICATION: finished
0812 08:01:43.373258 (+   390us) write_op.cc:301] APPLY: starting
0812 08:01:43.373297 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:43.373362 (+    65us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:43.373363 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:43.374937 (+  1574us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:43.374980 (+    43us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:43.374984 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:43.375088 (+   104us) log.cc:844] Serialized 319 byte log entry
0812 08:01:43.375173 (+    85us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:43.375216 (+    43us) write_op.cc:454] Released schema lock
0812 08:01:43.375226 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":284,"num_ops":50,"prepare.queue_time_us":21152630,"prepare.run_cpu_time_us":1434,"prepare.run_wall_time_us":1021627,"raft.queue_time_us":266,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":177,"replication_time_us":1285245,"row_lock_wait_count":1,"row_lock_wait_us":1020423,"thread_start_us":109,"threads_started":1}]]}
W20260812 08:01:43.377318 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1167ms of latency in SegmentAllocator::Sync()
W20260812 08:01:43.426301 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 22 similar messages]
W20260812 08:01:43.477475 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:43.491242 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 137) took 826 ms (client timeout 50 ms). Trace:
I20260812 08:01:43.491242 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.491415 32086 rpcz_store.cc:269] 0812 08:01:42.664843 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.664994 (+   151us) service_pool.cc:224] Handling call
0812 08:01:42.665243 (+   249us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:42.665530 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:42.665743 (+   213us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:42.665842 (+    99us) write_op.cc:270] Start()
0812 08:01:42.665869 (+    27us) write_op.cc:276] Timestamp: P: 1786521702087503 usec, L: 0
0812 08:01:42.666200 (+   331us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:42.666245 (+    45us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:42.666251 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:42.666266 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:43.491072 (+824806us) raft_consensus.cc:1685] finished
0812 08:01:43.491074 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.491209 (+   135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:42.666091 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:42.666212 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:42.666219 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:42.666221 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:42.666460 (+   239us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:42.666470 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:42.666472 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:42.667532 (+  1060us) tablet.cc:701] Row locks acquired
0812 08:01:42.667533 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":144,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":1508,"row_lock_wait_count":1,"row_lock_wait_us":752,"spinlock_wait_cycles":11520,"thread_start_us":80,"threads_started":1}]]}
W20260812 08:01:43.491672 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 140) took 259 ms (client timeout 50 ms). Trace:
I20260812 08:01:43.491725 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.491716 32090 rpcz_store.cc:269] 0812 08:01:43.231996 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.232129 (+   133us) service_pool.cc:224] Handling call
0812 08:01:43.491194 (+259065us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:43.491472 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.491475 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.491487 (+    12us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.491490 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.491504 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.491634 (+   130us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:43.491659 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595310336}
I20260812 08:01:43.492030 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.28]   Dedup: 1.28->[]
W20260812 08:01:43.492074 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 139) took 311 ms (client timeout 50 ms). Trace:
W20260812 08:01:43.492136 32088 rpcz_store.cc:269] 0812 08:01:43.180829 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.180962 (+   133us) service_pool.cc:224] Handling call
0812 08:01:43.491650 (+310688us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:43.491872 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.491877 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.491887 (+    10us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.491893 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.491906 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.492024 (+   118us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:43.492059 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713977600}
W20260812 08:01:43.492300 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 138) took 776 ms (client timeout 50 ms). Trace:
W20260812 08:01:43.492522 32089 rpcz_store.cc:269] 0812 08:01:42.715992 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.716155 (+   163us) service_pool.cc:224] Handling call
0812 08:01:43.492005 (+775850us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:43.492137 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 27
0812 08:01:43.492141 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:43.492149 (+     8us) raft_consensus.cc:1645] Marking committed up to 27
0812 08:01:43.492153 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.492163 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:43.492252 (+    89us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:43.492288 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1783713152}
I20260812 08:01:43.684185 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.28->[1.29-1.29]
W20260812 08:01:43.685788 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 896ms of latency in SegmentAllocator::Sync()
W20260812 08:01:44.376669 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9600
W20260812 08:01:44.544931 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 176) took 1168 ms (client timeout 50 ms). Trace:
I20260812 08:01:44.544981 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.545054 31936 rpcz_store.cc:269] 0812 08:01:43.376170 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.376260 (+    90us) service_pool.cc:224] Handling call
0812 08:01:43.376409 (+   149us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:43.376492 (+    83us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:43.376495 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:43.376628 (+   133us) write_op.cc:270] Start()
0812 08:01:43.376657 (+    29us) write_op.cc:276] Timestamp: P: 1786521703375271 usec, L: 0
0812 08:01:43.376996 (+   339us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:43.377033 (+    37us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:43.377040 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.377054 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:44.544769 (+1167715us) raft_consensus.cc:1685] finished
0812 08:01:44.544772 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.544901 (+   129us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:43.377050 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:43.377272 (+   222us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:43.377277 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:43.377280 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:43.377644 (+   364us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:43.377654 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:43.377657 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:43.377827 (+   170us) tablet.cc:701] Row locks acquired
0812 08:01:43.377828 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":316,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":820,"thread_start_us":100,"threads_started":1}]]}
I20260812 08:01:44.545379 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29]   Dedup: 1.29->[]
I20260812 08:01:44.545563 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.545642 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 177) took 1118 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.545687 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 179) took 707 ms (client timeout 50 ms). Trace:
I20260812 08:01:44.545800 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.545730 31938 rpcz_store.cc:269] 0812 08:01:43.838166 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.838310 (+   144us) service_pool.cc:224] Handling call
0812 08:01:44.544928 (+706618us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.545253 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.545256 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.545267 (+    11us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.545270 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.545280 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.545436 (+   156us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:44.545470 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1624567168}
W20260812 08:01:44.545900 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 178) took 758 ms (client timeout 50 ms). Trace:
I20260812 08:01:44.545972 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.546044 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 180) took 173 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.545955 31942 rpcz_store.cc:269] 0812 08:01:43.787340 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.787734 (+   394us) service_pool.cc:224] Handling call
0812 08:01:44.545538 (+757804us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.545675 (+   137us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.545689 (+    14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.545700 (+    11us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.545708 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.545725 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.545859 (+   134us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:44.545886 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1742284672}
W20260812 08:01:44.546116 31939 rpcz_store.cc:269] 0812 08:01:44.373065 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.373248 (+   183us) service_pool.cc:224] Handling call
0812 08:01:44.545781 (+172533us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.545876 (+    95us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.545879 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.545886 (+     7us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.545889 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.545897 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.546009 (+   112us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:44.546034 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":396358912}
W20260812 08:01:44.546299 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 181) took 122 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.545703 31935 rpcz_store.cc:269] 0812 08:01:43.427424 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.427525 (+   101us) service_pool.cc:224] Handling call
0812 08:01:44.545352 (+1117827us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.545479 (+   127us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.545482 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.545489 (+     7us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.545492 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.545500 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.545607 (+   107us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:44.545631 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2570322688}
W20260812 08:01:44.546362 31941 rpcz_store.cc:269] 0812 08:01:44.424020 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.424100 (+    80us) service_pool.cc:224] Handling call
0812 08:01:44.545948 (+121848us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.546095 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.546101 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.546111 (+    10us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.546115 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.546127 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.546241 (+   114us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:44.546267 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279776640}
W20260812 08:01:44.582531 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 141) took 898 ms (client timeout 50 ms). Trace:
I20260812 08:01:44.582531 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.582772 32089 rpcz_store.cc:269] 0812 08:01:43.683722 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.683809 (+    87us) service_pool.cc:224] Handling call
0812 08:01:43.684093 (+   284us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:43.684402 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:43.684659 (+   257us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:43.684835 (+   176us) write_op.cc:270] Start()
0812 08:01:43.684880 (+    45us) write_op.cc:276] Timestamp: P: 1786521703375271 usec, L: 0
0812 08:01:43.685483 (+   603us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:43.685534 (+    51us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:43.685541 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:43.685558 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:44.582352 (+896794us) raft_consensus.cc:1685] finished
0812 08:01:44.582355 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.582496 (+   141us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:43.685386 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:43.685520 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:43.685525 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:43.685527 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:43.685779 (+   252us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:43.685789 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:43.685791 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:43.686898 (+  1107us) tablet.cc:701] Row locks acquired
0812 08:01:43.686900 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":390,"prepare.run_cpu_time_us":872,"prepare.run_wall_time_us":1579,"row_lock_wait_count":1,"row_lock_wait_us":788,"thread_start_us":271,"threads_started":1}]]}
I20260812 08:01:44.583036 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.583061 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 142) took 848 ms (client timeout 50 ms). Trace:
I20260812 08:01:44.583489 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.583407 32088 rpcz_store.cc:269] 0812 08:01:43.734987 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.735124 (+   137us) service_pool.cc:224] Handling call
0812 08:01:44.582476 (+847352us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:44.582862 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.582867 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.582881 (+    14us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.582885 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.582899 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.583013 (+   114us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:44.583044 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1948140416}
I20260812 08:01:44.583784 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.584033 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 144) took 419 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.584170 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 145) took 94 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.584112 32086 rpcz_store.cc:269] 0812 08:01:44.164938 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.165111 (+   173us) service_pool.cc:224] Handling call
0812 08:01:44.583461 (+418350us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:44.583637 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.583643 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.583666 (+    23us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.583670 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.583683 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.583989 (+   306us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:44.584020 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961414400}
W20260812 08:01:44.584223 32085 rpcz_store.cc:269] 0812 08:01:44.490066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.490221 (+   155us) service_pool.cc:224] Handling call
0812 08:01:44.583753 (+ 93532us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:44.583987 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.583990 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.583998 (+     8us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.584002 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.584012 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.584129 (+   117us) spinlock_profiling.cc:243] Waited 93.3 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:44.584150 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214572928}
I20260812 08:01:44.584113 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.25->[1.26-1.29]   Dedup: 1.29->[]
W20260812 08:01:44.584527 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 143) took 471 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.584774 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 146) took 43 ms (client timeout 50 ms). Trace:
W20260812 08:01:44.584682 32090 rpcz_store.cc:269] 0812 08:01:44.113595 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.113740 (+   145us) service_pool.cc:224] Handling call
0812 08:01:44.582998 (+469258us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:44.583391 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.583395 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.583404 (+     9us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.583412 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.583426 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.584486 (+  1060us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:44.584513 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1078591744}
W20260812 08:01:44.584844 32084 rpcz_store.cc:269] 0812 08:01:44.540970 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.541101 (+   131us) service_pool.cc:224] Handling call
0812 08:01:44.584063 (+ 42962us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:44.584491 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 28
0812 08:01:44.584493 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:44.584500 (+     7us) raft_consensus.cc:1645] Marking committed up to 28
0812 08:01:44.584504 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.584513 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:44.584725 (+   212us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:44.584760 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98051840}
W20260812 08:01:44.588969 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1103ms of latency in SegmentAllocator::Sync()
I20260812 08:01:44.589767 32587 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=27, attempt_no=0}) took 23270 ms. Trace:
I20260812 08:01:44.589864 32587 rpcz_store.cc:276] 0812 08:01:21.319268 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:21.319385 (+   117us) service_pool.cc:224] Handling call
0812 08:01:44.589731 (+23270346us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:42.087863 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:42.088011 (+   148us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:42.088016 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:42.088018 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:42.088334 (+   316us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:42.088342 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:42.088345 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:43.375234 (+1286889us) lock_manager.cc:598] Waited 1286560us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:43.375257 (+    23us) tablet.cc:701] Row locks acquired
0812 08:01:43.375260 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:01:43.375330 (+    70us) write_op.cc:270] Start()
0812 08:01:43.375359 (+    29us) write_op.cc:276] Timestamp: P: 1786521703375271 usec, L: 0
0812 08:01:43.375361 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:43.375524 (+   163us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:44.585841 (+1210317us) op_driver.cc:464] REPLICATION: finished
0812 08:01:44.586427 (+   586us) write_op.cc:301] APPLY: starting
0812 08:01:44.586464 (+    37us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:44.586531 (+    67us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:44.586533 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:44.588075 (+  1542us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:44.588256 (+   181us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:44.588262 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:44.588392 (+   130us) log.cc:844] Serialized 319 byte log entry
0812 08:01:44.588463 (+    71us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:44.589617 (+  1154us) write_op.cc:454] Released schema lock
0812 08:01:44.589623 (+     6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":525,"num_ops":50,"prepare.queue_time_us":20768042,"prepare.run_cpu_time_us":1428,"prepare.run_wall_time_us":1287779,"raft.queue_time_us":131,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":553,"replication_time_us":1210445,"row_lock_wait_count":1,"row_lock_wait_us":1286560,"spinlock_wait_cycles":105984,"thread_start_us":229,"threads_started":1}]]}
W20260812 08:01:44.590834 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1060ms of latency in SegmentAllocator::Sync()
W20260812 08:01:44.691111 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:44.756232 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.30]   Dedup: 1.29->[1.30-1.30]
W20260812 08:01:44.757236 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1043ms of latency in SegmentAllocator::Sync()
W20260812 08:01:45.589713 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8dc0
I20260812 08:01:45.651503 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.651503 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 148) took 1062 ms (client timeout 50 ms). Trace:
W20260812 08:01:45.651711 32084 rpcz_store.cc:269] 0812 08:01:44.589618 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.589705 (+    87us) service_pool.cc:224] Handling call
0812 08:01:44.589867 (+   162us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:44.589952 (+    85us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:44.589955 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:44.590085 (+   130us) write_op.cc:270] Start()
0812 08:01:44.590110 (+    25us) write_op.cc:276] Timestamp: P: 1786521704588533 usec, L: 0
0812 08:01:44.590442 (+   332us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:44.590483 (+    41us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:44.590492 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.590506 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:45.651328 (+1060822us) raft_consensus.cc:1685] finished
0812 08:01:45.651331 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.651477 (+   146us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:44.590613 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:44.590745 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:44.590750 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:44.590752 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:44.591080 (+   328us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:44.591089 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:44.591092 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:44.591309 (+   217us) tablet.cc:701] Row locks acquired
0812 08:01:44.591310 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":384,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":796,"thread_start_us":76,"threads_started":1}]]}
I20260812 08:01:45.651921 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.651961 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 151) took 460 ms (client timeout 50 ms). Trace:
W20260812 08:01:45.652285 32086 rpcz_store.cc:269] 0812 08:01:45.191349 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.191467 (+   118us) service_pool.cc:224] Handling call
0812 08:01:45.651448 (+459981us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:45.651748 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.651751 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.651764 (+    13us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.651768 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.651780 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.651921 (+   141us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:45.651947 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057380224}
I20260812 08:01:45.652499 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.652535 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 149) took 1011 ms (client timeout 50 ms). Trace:
W20260812 08:01:45.652684 32090 rpcz_store.cc:269] 0812 08:01:44.640813 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.641044 (+   231us) service_pool.cc:224] Handling call
0812 08:01:45.651878 (+1010834us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:45.652301 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.652305 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.652317 (+    12us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.652323 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.652340 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.652480 (+   140us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:45.652519 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2324206464}
W20260812 08:01:45.652871 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 150) took 512 ms (client timeout 50 ms). Trace:
W20260812 08:01:45.652972 32085 rpcz_store.cc:269] 0812 08:01:45.140282 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.140379 (+    97us) service_pool.cc:224] Handling call
0812 08:01:45.652394 (+512015us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:45.652700 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.652707 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.652717 (+    10us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.652721 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.652732 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.652836 (+   104us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:45.652859 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1177111936}
I20260812 08:01:45.697198 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.701540 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1015ms of latency in SegmentAllocator::Sync()
I20260812 08:01:45.701933 32597 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=28, attempt_no=0}) took 23388 ms. Trace:
I20260812 08:01:45.702021 32597 rpcz_store.cc:276] 0812 08:01:22.313611 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:22.313717 (+   106us) service_pool.cc:224] Handling call
0812 08:01:45.701917 (+23388200us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:43.375683 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:43.375803 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:43.375808 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:43.375809 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:43.376173 (+   364us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:43.376184 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:43.376189 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:44.588505 (+1212316us) lock_manager.cc:598] Waited 1212020us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:44.588518 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:44.588520 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:44.588567 (+    47us) write_op.cc:270] Start()
0812 08:01:44.588596 (+    29us) write_op.cc:276] Timestamp: P: 1786521704588533 usec, L: 0
0812 08:01:44.588598 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:44.588734 (+   136us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:45.698565 (+1109831us) op_driver.cc:464] REPLICATION: finished
0812 08:01:45.698964 (+   399us) write_op.cc:301] APPLY: starting
0812 08:01:45.699013 (+    49us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:45.699106 (+    93us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:45.699108 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:45.700677 (+  1569us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:45.700739 (+    62us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:45.700745 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:45.700883 (+   138us) log.cc:844] Serialized 319 byte log entry
0812 08:01:45.700955 (+    72us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:45.701823 (+   868us) write_op.cc:454] Released schema lock
0812 08:01:45.701833 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":260,"num_ops":50,"prepare.queue_time_us":21061584,"prepare.run_cpu_time_us":1403,"prepare.run_wall_time_us":1213180,"raft.queue_time_us":134,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"replication_time_us":1109930,"row_lock_wait_count":1,"row_lock_wait_us":1212020,"spinlock_wait_cycles":384,"thread_start_us":111,"threads_started":1}]]}
W20260812 08:01:45.702857 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 940ms of latency in SegmentAllocator::Sync()
W20260812 08:01:45.801013 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 182) took 1045 ms (client timeout 50 ms). Trace:
I20260812 08:01:45.801067 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.801167 31941 rpcz_store.cc:269] 0812 08:01:44.755791 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.755947 (+   156us) service_pool.cc:224] Handling call
0812 08:01:44.756164 (+   217us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:44.756388 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:44.756563 (+   175us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:44.756642 (+    79us) write_op.cc:270] Start()
0812 08:01:44.756668 (+    26us) write_op.cc:276] Timestamp: P: 1786521704588533 usec, L: 0
0812 08:01:44.757022 (+   354us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:44.757342 (+   320us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:44.757349 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:44.757364 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:45.800886 (+1043522us) raft_consensus.cc:1685] finished
0812 08:01:45.800888 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.800986 (+    98us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:44.756921 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:44.757043 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:44.757048 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:44.757050 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:44.757396 (+   346us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:44.757409 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:44.757412 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:44.758353 (+   941us) tablet.cc:701] Row locks acquired
0812 08:01:44.758355 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":176,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":1508,"row_lock_wait_count":1,"row_lock_wait_us":742,"thread_start_us":105,"threads_started":1}]]}
W20260812 08:01:45.801499 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 184) took 469 ms (client timeout 50 ms). Trace:
I20260812 08:01:45.801594 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.801640 31939 rpcz_store.cc:269] 0812 08:01:45.331691 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.331767 (+    76us) service_pool.cc:224] Handling call
0812 08:01:45.801018 (+469251us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:45.801283 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.801290 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.801306 (+    16us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.801314 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.801332 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.801452 (+   120us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:45.801485 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1078663424}
W20260812 08:01:45.801885 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 185) took 419 ms (client timeout 50 ms). Trace:
I20260812 08:01:45.801971 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.30]   Dedup: 1.30->[]
W20260812 08:01:45.802042 31942 rpcz_store.cc:269] 0812 08:01:45.382770 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.382871 (+   101us) service_pool.cc:224] Handling call
0812 08:01:45.801554 (+418683us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:45.801691 (+   137us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.801694 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.801702 (+     8us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.801705 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.801717 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.801844 (+   127us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:45.801872 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":962386816}
W20260812 08:01:45.802323 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 183) took 995 ms (client timeout 50 ms). Trace:
W20260812 08:01:45.802435 31935 rpcz_store.cc:269] 0812 08:01:44.806783 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.806876 (+    93us) service_pool.cc:224] Handling call
0812 08:01:45.801943 (+995067us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:45.802112 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 29
0812 08:01:45.802115 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:45.802124 (+     9us) raft_consensus.cc:1645] Marking committed up to 29
0812 08:01:45.802145 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.802161 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:45.802286 (+   125us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:45.802312 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2287979264}
I20260812 08:01:45.802557 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.31]   Dedup: 1.30->[1.31-1.31]
W20260812 08:01:45.803542 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:45.803719 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1078ms of latency in SegmentAllocator::Sync()
W20260812 08:01:45.903553 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:01:46.643853 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 154) took 942 ms (client timeout 50 ms). Trace:
I20260812 08:01:46.644117 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.643973 32085 rpcz_store.cc:269] 0812 08:01:45.701864 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.701912 (+    48us) service_pool.cc:224] Handling call
0812 08:01:45.702066 (+   154us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:45.702154 (+    88us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:45.702157 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:45.702254 (+    97us) write_op.cc:270] Start()
0812 08:01:45.702278 (+    24us) write_op.cc:276] Timestamp: P: 1786521705701027 usec, L: 0
0812 08:01:45.702633 (+   355us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:45.702674 (+    41us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:45.702681 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.702696 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:46.643591 (+940895us) raft_consensus.cc:1685] finished
0812 08:01:46.643595 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.643809 (+   214us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:45.702685 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:45.702896 (+   211us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:45.702900 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:45.702902 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:45.703173 (+   271us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:45.703187 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:45.703191 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:45.703406 (+   215us) tablet.cc:701] Row locks acquired
0812 08:01:45.703407 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":303,"prepare.run_cpu_time_us":789,"prepare.run_wall_time_us":788,"thread_start_us":81,"threads_started":1}]]}
I20260812 08:01:46.644444 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.644466 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 155) took 891 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.644610 32090 rpcz_store.cc:269] 0812 08:01:45.753103 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.753254 (+   151us) service_pool.cc:224] Handling call
0812 08:01:46.644038 (+890784us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:46.644283 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.644285 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.644297 (+    12us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.644301 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.644314 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.644406 (+    92us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:46.644436 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2047777536}
I20260812 08:01:46.644694 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.644800 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 156) took 467 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.644876 32086 rpcz_store.cc:269] 0812 08:01:46.177764 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.177848 (+    84us) service_pool.cc:224] Handling call
0812 08:01:46.644391 (+466543us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:46.644577 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.644581 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.644589 (+     8us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.644593 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.644606 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.644755 (+   149us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:46.644785 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1072367872}
W20260812 08:01:46.645021 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 157) took 416 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.645061 32084 rpcz_store.cc:269] 0812 08:01:46.229040 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.229189 (+   149us) service_pool.cc:224] Handling call
0812 08:01:46.644667 (+415478us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:46.644854 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.644859 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.644866 (+     7us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.644871 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.644881 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.644985 (+   104us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:46.645011 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954755328}
I20260812 08:01:46.698405 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.702767 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8b00
I20260812 08:01:46.720263 32605 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=29, attempt_no=0}) took 23354 ms. Trace:
I20260812 08:01:46.720477 32605 rpcz_store.cc:276] 0812 08:01:23.365423 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:23.365516 (+    93us) service_pool.cc:224] Handling call
0812 08:01:46.720228 (+23354712us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:44.588891 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:44.589031 (+   140us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:44.589036 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:44.589038 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:44.589373 (+   335us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:44.589381 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:44.589383 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:45.700994 (+1111611us) lock_manager.cc:598] Waited 1111413us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:45.701008 (+    14us) tablet.cc:701] Row locks acquired
0812 08:01:45.701009 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:01:45.701057 (+    48us) write_op.cc:270] Start()
0812 08:01:45.701080 (+    23us) write_op.cc:276] Timestamp: P: 1786521705701027 usec, L: 0
0812 08:01:45.701082 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:45.701228 (+   146us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:46.717275 (+1016047us) op_driver.cc:464] REPLICATION: finished
0812 08:01:46.717705 (+   430us) write_op.cc:301] APPLY: starting
0812 08:01:46.717753 (+    48us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:46.717838 (+    85us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:46.717840 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:46.719814 (+  1974us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:46.719875 (+    61us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:46.719881 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:46.720023 (+   142us) log.cc:844] Serialized 319 byte log entry
0812 08:01:46.720100 (+    77us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:46.720134 (+    34us) write_op.cc:454] Released schema lock
0812 08:01:46.720143 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":291,"num_ops":50,"prepare.queue_time_us":21222943,"prepare.run_cpu_time_us":1287,"prepare.run_wall_time_us":1112433,"raft.queue_time_us":316,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":198,"replication_time_us":1016137,"row_lock_wait_count":1,"row_lock_wait_us":1111413,"thread_start_us":123,"threads_started":1}]]}
W20260812 08:01:46.721472 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 909ms of latency in SegmentAllocator::Sync()
W20260812 08:01:46.722986 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 860ms of latency in SegmentAllocator::Sync()
W20260812 08:01:46.823946 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:46.882349 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 186) took 1080 ms (client timeout 50 ms). Trace:
I20260812 08:01:46.882349 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.882544 31942 rpcz_store.cc:269] 0812 08:01:45.802219 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.802295 (+    76us) service_pool.cc:224] Handling call
0812 08:01:45.802507 (+   212us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:45.802710 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:45.802970 (+   260us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:45.803094 (+   124us) write_op.cc:270] Start()
0812 08:01:45.803122 (+    28us) write_op.cc:276] Timestamp: P: 1786521705701027 usec, L: 0
0812 08:01:45.803430 (+   308us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:45.803469 (+    39us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:45.803474 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:45.803489 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:46.882177 (+1078688us) raft_consensus.cc:1685] finished
0812 08:01:46.882180 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.882317 (+   137us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:45.803351 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:45.803479 (+   128us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:45.803483 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:45.803484 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:45.803949 (+   465us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:45.803960 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:45.803963 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:45.804745 (+   782us) tablet.cc:701] Row locks acquired
0812 08:01:45.804748 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":1231,"prepare.run_wall_time_us":1514,"row_lock_wait_count":1,"row_lock_wait_us":378,"thread_start_us":84,"threads_started":1}]]}
I20260812 08:01:46.882812 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.882863 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 187) took 1029 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.882916 31935 rpcz_store.cc:269] 0812 08:01:45.853377 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.853499 (+   122us) service_pool.cc:224] Handling call
0812 08:01:46.882297 (+1028798us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:46.882642 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.882646 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.882659 (+    13us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.882663 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.882703 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.882821 (+   118us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:46.882849 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2365394816}
I20260812 08:01:46.883291 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.31]   Dedup: 1.31->[]
W20260812 08:01:46.883352 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 188) took 589 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.883401 31939 rpcz_store.cc:269] 0812 08:01:46.294338 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.294504 (+   166us) service_pool.cc:224] Handling call
0812 08:01:46.882769 (+588265us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:46.883049 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.883054 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.883069 (+    15us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.883076 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.883100 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.883260 (+   160us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:46.883335 (+    75us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1352300928}
W20260812 08:01:46.883651 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 189) took 538 ms (client timeout 50 ms). Trace:
W20260812 08:01:46.883704 31941 rpcz_store.cc:269] 0812 08:01:46.345310 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.345438 (+   128us) service_pool.cc:224] Handling call
0812 08:01:46.883261 (+537823us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:46.883411 (+   150us) raft_consensus.cc:1534] Early marking committed up to index 30
0812 08:01:46.883415 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:46.883424 (+     9us) raft_consensus.cc:1645] Marking committed up to 30
0812 08:01:46.883428 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.883473 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:46.883611 (+   138us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:46.883638 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1236431616}
I20260812 08:01:46.883899 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.32]   Dedup: 1.31->[1.32-1.32]
W20260812 08:01:46.885076 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1111ms of latency in SegmentAllocator::Sync()
W20260812 08:01:47.583631 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 160) took 861 ms (client timeout 50 ms). Trace:
I20260812 08:01:47.583693 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.583770 32090 rpcz_store.cc:269] 0812 08:01:46.721820 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.721923 (+   103us) service_pool.cc:224] Handling call
0812 08:01:46.722085 (+   162us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:46.722185 (+   100us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:46.722188 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:46.722291 (+   103us) write_op.cc:270] Start()
0812 08:01:46.722313 (+    22us) write_op.cc:276] Timestamp: P: 1786521706720215 usec, L: 0
0812 08:01:46.722697 (+   384us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:46.722739 (+    42us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:46.722745 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.722759 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:47.583500 (+860741us) raft_consensus.cc:1685] finished
0812 08:01:47.583503 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.583598 (+    95us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:46.722639 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:46.722769 (+   130us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:46.722775 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:46.722778 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:46.722991 (+   213us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:46.723005 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:46.723007 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:46.723381 (+   374us) tablet.cc:701] Row locks acquired
0812 08:01:46.723383 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":246,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":807,"thread_start_us":132,"threads_started":1}]]}
I20260812 08:01:47.584055 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.584168 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 163) took 379 ms (client timeout 50 ms). Trace:
W20260812 08:01:47.584267 32085 rpcz_store.cc:269] 0812 08:01:47.205394 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.205773 (+   379us) service_pool.cc:224] Handling call
0812 08:01:47.583646 (+377873us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:47.583876 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:47.583881 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:47.583896 (+    15us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:47.583904 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.583922 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.584104 (+   182us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:47.584149 (+    45us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":868416768}
W20260812 08:01:47.584398 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 161) took 810 ms (client timeout 50 ms). Trace:
I20260812 08:01:47.584458 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.584563 32084 rpcz_store.cc:269] 0812 08:01:46.773508 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.773600 (+    92us) service_pool.cc:224] Handling call
0812 08:01:47.583994 (+810394us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:47.584161 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:47.584164 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:47.584173 (+     9us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:47.584178 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.584256 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.584354 (+    98us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:47.584383 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1863277824}
I20260812 08:01:47.584836 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.584863 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 162) took 432 ms (client timeout 50 ms). Trace:
W20260812 08:01:47.584949 32086 rpcz_store.cc:269] 0812 08:01:47.152583 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.152743 (+   160us) service_pool.cc:224] Handling call
0812 08:01:47.584424 (+431681us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:47.584661 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:47.584665 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:47.584673 (+     8us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:47.584693 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.584704 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.584822 (+   118us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:47.584851 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":992406400}
W20260812 08:01:47.634459 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 888ms of latency in SegmentAllocator::Sync()
I20260812 08:01:47.634979 32613 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=30, attempt_no=0}) took 22807 ms. Trace:
I20260812 08:01:47.635118 32613 rpcz_store.cc:276] 0812 08:01:24.827804 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:24.827940 (+   136us) service_pool.cc:224] Handling call
0812 08:01:47.634961 (+22807021us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:45.701354 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:45.701470 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:45.701475 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:45.701477 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:45.701823 (+   346us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:45.701832 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:45.701835 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:46.720153 (+1018318us) lock_manager.cc:598] Waited 1017527us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:46.720168 (+    15us) tablet.cc:701] Row locks acquired
0812 08:01:46.720170 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:46.720251 (+    81us) write_op.cc:270] Start()
0812 08:01:46.720281 (+    30us) write_op.cc:276] Timestamp: P: 1786521706720215 usec, L: 0
0812 08:01:46.720283 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:46.720485 (+   202us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:47.631134 (+910649us) op_driver.cc:464] REPLICATION: finished
0812 08:01:47.631558 (+   424us) write_op.cc:301] APPLY: starting
0812 08:01:47.631604 (+    46us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:47.631672 (+    68us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:47.631674 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:47.633568 (+  1894us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:47.633660 (+    92us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:47.633666 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:47.633807 (+   141us) log.cc:844] Serialized 319 byte log entry
0812 08:01:47.633890 (+    83us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:47.634857 (+   967us) write_op.cc:454] Released schema lock
0812 08:01:47.634866 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":318,"num_ops":50,"prepare.queue_time_us":20873013,"prepare.run_cpu_time_us":1919,"prepare.run_wall_time_us":1019211,"raft.queue_time_us":1069,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":910802,"row_lock_wait_count":1,"row_lock_wait_us":1017527,"spinlock_wait_cycles":56192,"thread_start_us":124,"threads_started":1}]]}
W20260812 08:01:47.636175 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 970ms of latency in SegmentAllocator::Sync()
W20260812 08:01:47.736498 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:47.996953 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.997263 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 191) took 1219 ms (client timeout 50 ms). Trace:
I20260812 08:01:47.997546 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.32]   Dedup: 1.32->[]
W20260812 08:01:47.997330 31936 rpcz_store.cc:269] 0812 08:01:46.777381 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.777512 (+   131us) service_pool.cc:224] Handling call
0812 08:01:46.883801 (+106289us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:46.884065 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:46.884272 (+   207us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:46.884400 (+   128us) write_op.cc:270] Start()
0812 08:01:46.884426 (+    26us) write_op.cc:276] Timestamp: P: 1786521706720215 usec, L: 0
0812 08:01:46.884778 (+   352us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:46.884821 (+    43us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:46.884827 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:46.884838 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:47.996712 (+1111874us) raft_consensus.cc:1685] finished
0812 08:01:47.996715 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.997165 (+   450us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:47.997239 (+    74us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:46.884692 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:46.884835 (+   143us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:46.884840 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:46.884841 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:46.885310 (+   469us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:46.885321 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:46.885323 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:46.886874 (+  1551us) tablet.cc:701] Row locks acquired
0812 08:01:46.886877 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":243770240,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":1331,"prepare.run_wall_time_us":2280,"row_lock_wait_count":1,"row_lock_wait_us":1308,"thread_start_us":102,"threads_started":1}]]}
W20260812 08:01:47.997561 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 190) took 1271 ms (client timeout 50 ms). Trace:
W20260812 08:01:47.997726 31938 rpcz_store.cc:269] 0812 08:01:46.726323 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.726404 (+    81us) service_pool.cc:224] Handling call
0812 08:01:47.996875 (+1270471us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:47.997348 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:47.997352 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:47.997364 (+    12us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:47.997368 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.997382 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.997526 (+   144us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:47.997550 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2920346880}
W20260812 08:01:47.997926 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 193) took 783 ms (client timeout 50 ms). Trace:
I20260812 08:01:47.997980 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.32->[1.33-1.33]
W20260812 08:01:47.998018 31939 rpcz_store.cc:269] 0812 08:01:47.214106 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.214367 (+   261us) service_pool.cc:224] Handling call
0812 08:01:47.997509 (+783142us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:47.997728 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:47.997731 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:47.997741 (+    10us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:47.997746 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.997761 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:47.997878 (+   117us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:47.997911 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1800407296}
W20260812 08:01:47.999964 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1121ms of latency in SegmentAllocator::Sync()
W20260812 08:01:48.432148 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 11: this message will repeat every 5th retry.
W20260812 08:01:48.607064 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 166) took 971 ms (client timeout 50 ms). Trace:
I20260812 08:01:48.607072 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.32->[1.33-1.33]   Dedup: 1.33->[]
I20260812 08:01:48.607637 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.32->[1.33-1.33]   Dedup: 1.33->[]
W20260812 08:01:48.607429 32088 rpcz_store.cc:269] 0812 08:01:47.634944 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.634993 (+    49us) service_pool.cc:224] Handling call
0812 08:01:47.635173 (+   180us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:47.635404 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:47.635407 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:47.635538 (+   131us) write_op.cc:270] Start()
0812 08:01:47.635567 (+    29us) write_op.cc:276] Timestamp: P: 1786521707633958 usec, L: 0
0812 08:01:47.635940 (+   373us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:47.635974 (+    34us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:47.635980 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.635995 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:48.606737 (+970742us) raft_consensus.cc:1685] finished
0812 08:01:48.606740 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:48.606861 (+   121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:47.635947 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:47.636072 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:47.636076 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:47.636078 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:47.636458 (+   380us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:47.636472 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:47.636476 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:47.636792 (+   316us) tablet.cc:701] Row locks acquired
0812 08:01:47.636794 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":274,"prepare.run_cpu_time_us":932,"prepare.run_wall_time_us":932,"thread_start_us":126,"threads_started":1}]]}
W20260812 08:01:48.607877 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 167) took 921 ms (client timeout 50 ms). Trace:
W20260812 08:01:48.607944 32086 rpcz_store.cc:269] 0812 08:01:47.686661 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.686878 (+   217us) service_pool.cc:224] Handling call
0812 08:01:48.606826 (+919948us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:48.607497 (+   671us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:48.607501 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:48.607513 (+    12us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:48.607518 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:48.607531 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:48.607821 (+   290us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:48.607860 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2115155712}
W20260812 08:01:48.608445 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 169) took 378 ms (client timeout 50 ms). Trace:
W20260812 08:01:48.608587 32085 rpcz_store.cc:269] 0812 08:01:48.230488 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.230877 (+   389us) service_pool.cc:224] Handling call
0812 08:01:48.607602 (+376725us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:48.607764 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:48.607794 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:48.607813 (+    19us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:48.607820 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:48.607835 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:48.608386 (+   551us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:48.608423 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":865916160}
I20260812 08:01:48.608930 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.32->[1.33-1.33]   Dedup: 1.33->[]
W20260812 08:01:48.609272 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 168) took 429 ms (client timeout 50 ms). Trace:
W20260812 08:01:48.609341 32084 rpcz_store.cc:269] 0812 08:01:48.179527 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.179766 (+   239us) service_pool.cc:224] Handling call
0812 08:01:48.608896 (+429130us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:48.609075 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:48.609079 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:48.609088 (+     9us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:48.609092 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:48.609104 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:48.609237 (+   133us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:48.609261 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986425216}
W20260812 08:01:48.635344 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8580
I20260812 08:01:48.770813 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.32->[1.33-1.33]   Dedup: 1.33->[]
I20260812 08:01:48.775115 32622 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=31, attempt_no=0}) took 22980 ms. Trace:
W20260812 08:01:48.775537 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 978ms of latency in SegmentAllocator::Sync()
I20260812 08:01:48.775293 32622 rpcz_store.cc:276] 0812 08:01:25.794928 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:25.795004 (+    76us) service_pool.cc:224] Handling call
0812 08:01:48.775091 (+22980087us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:46.720597 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:46.720832 (+   235us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:46.720837 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:46.720839 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:46.721200 (+   361us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:46.721215 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:46.721219 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:47.633927 (+912708us) lock_manager.cc:598] Waited 912426us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:47.633940 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:47.633942 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:47.633989 (+    47us) write_op.cc:270] Start()
0812 08:01:47.634012 (+    23us) write_op.cc:276] Timestamp: P: 1786521707633958 usec, L: 0
0812 08:01:47.634014 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:47.634222 (+   208us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:48.772096 (+1137874us) op_driver.cc:464] REPLICATION: finished
0812 08:01:48.772340 (+   244us) write_op.cc:301] APPLY: starting
0812 08:01:48.772908 (+   568us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:48.773004 (+    96us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:48.773006 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:48.774703 (+  1697us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:48.774751 (+    48us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:48.774755 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:48.774865 (+   110us) log.cc:844] Serialized 319 byte log entry
0812 08:01:48.774962 (+    97us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:48.774998 (+    36us) write_op.cc:454] Released schema lock
0812 08:01:48.775007 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":50,"prepare.queue_time_us":20925186,"prepare.run_cpu_time_us":1328,"prepare.run_wall_time_us":913736,"raft.queue_time_us":261,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"replication_time_us":1138046,"row_lock_wait_count":1,"row_lock_wait_us":912426,"spinlock_wait_cycles":9472,"thread_start_us":71,"threads_started":1}]]}
W20260812 08:01:48.777365 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 919ms of latency in SegmentAllocator::Sync()
W20260812 08:01:48.799793 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 21 similar messages]
W20260812 08:01:48.878170 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:49.121902 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.33->[]
W20260812 08:01:49.122052 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 194) took 1382 ms (client timeout 50 ms). Trace:
I20260812 08:01:49.122287 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.32]   Dedup: 1.32->[]
W20260812 08:01:49.122326 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 195) took 1331 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.122162 31935 rpcz_store.cc:269] 0812 08:01:47.740071 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.740150 (+    79us) service_pool.cc:224] Handling call
0812 08:01:47.997897 (+257747us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:47.998094 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:47.998638 (+   544us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:47.998874 (+   236us) write_op.cc:270] Start()
0812 08:01:47.998911 (+    37us) write_op.cc:276] Timestamp: P: 1786521707633958 usec, L: 0
0812 08:01:47.999534 (+   623us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:47.999582 (+    48us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:47.999589 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:47.999602 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:49.121571 (+1121969us) raft_consensus.cc:1685] finished
0812 08:01:49.121574 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.121896 (+   322us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.121955 (+    59us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:47.999419 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:47.999552 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:47.999557 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:47.999559 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:47.999967 (+   408us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:47.999977 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:47.999979 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:48.001126 (+  1147us) tablet.cc:701] Row locks acquired
0812 08:01:48.001128 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":592233728,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":413,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":1977,"row_lock_wait_count":1,"row_lock_wait_us":901,"thread_start_us":341,"threads_started":1}]]}
W20260812 08:01:49.122404 31942 rpcz_store.cc:269] 0812 08:01:47.791295 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.791477 (+   182us) service_pool.cc:224] Handling call
0812 08:01:49.121845 (+1330368us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:49.122101 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:49.122108 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.122121 (+    13us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:49.122126 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.122143 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.122286 (+   143us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.122314 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3058843520}
W20260812 08:01:49.122604 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 192) took 1959 ms (client timeout 50 ms). Trace:
I20260812 08:01:49.122702 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.33->[]
W20260812 08:01:49.122666 31941 rpcz_store.cc:269] 0812 08:01:47.162867 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.162990 (+   123us) service_pool.cc:224] Handling call
0812 08:01:49.122246 (+1959256us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:49.122403 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 31
0812 08:01:49.122406 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.122414 (+     8us) raft_consensus.cc:1645] Marking committed up to 31
0812 08:01:49.122418 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.122429 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.122539 (+   110us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.122563 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4505132928}
W20260812 08:01:49.123006 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 197) took 741 ms (client timeout 50 ms). Trace:
I20260812 08:01:49.123006 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.33->[]
W20260812 08:01:49.123107 31936 rpcz_store.cc:269] 0812 08:01:48.382032 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.382275 (+   243us) service_pool.cc:224] Handling call
0812 08:01:49.122674 (+740399us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:49.122817 (+   143us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:49.122820 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.122829 (+     9us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:49.122837 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.122848 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.122967 (+   119us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.122994 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1702186368}
I20260812 08:01:49.123294 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.33->[]
W20260812 08:01:49.123443 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 198) took 374 ms (client timeout 50 ms). Trace:
I20260812 08:01:49.123683 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.33]   Dedup: 1.33->[]
W20260812 08:01:49.123754 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 196) took 792 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.123857 31938 rpcz_store.cc:269] 0812 08:01:48.749450 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.749588 (+   138us) service_pool.cc:224] Handling call
0812 08:01:49.122984 (+373396us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:49.123166 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:49.123169 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.123176 (+     7us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:49.123180 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.123190 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.123353 (+   163us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.123427 (+    74us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":858162560}
W20260812 08:01:49.123970 31939 rpcz_store.cc:269] 0812 08:01:48.330940 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.331020 (+    80us) service_pool.cc:224] Handling call
0812 08:01:49.123258 (+792238us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:49.123404 (+   146us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:49.123408 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.123415 (+     7us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:49.123419 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.123429 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.123706 (+   277us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.123739 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821426560}
W20260812 08:01:49.124140 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 199) took 323 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.124258 31937 rpcz_store.cc:269] 0812 08:01:48.800792 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.800878 (+    86us) service_pool.cc:224] Handling call
0812 08:01:49.123641 (+322763us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:01:49.123941 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 32
0812 08:01:49.123944 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.123954 (+    10us) raft_consensus.cc:1645] Marking committed up to 32
0812 08:01:49.123960 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.123973 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.124104 (+   131us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:49.124128 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":741699968}
I20260812 08:01:49.249750 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.34]   Dedup: 1.33->[1.34-1.34]
W20260812 08:01:49.250881 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 950ms of latency in SegmentAllocator::Sync()
W20260812 08:01:49.697012 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 172) took 921 ms (client timeout 50 ms). Trace:
I20260812 08:01:49.697037 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.34]   Dedup: 1.34->[]
W20260812 08:01:49.697190 32084 rpcz_store.cc:269] 0812 08:01:48.775951 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.776029 (+    78us) service_pool.cc:224] Handling call
0812 08:01:48.776207 (+   178us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:48.776328 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:48.776332 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:48.776466 (+   134us) write_op.cc:270] Start()
0812 08:01:48.776495 (+    29us) write_op.cc:276] Timestamp: P: 1786521708775064 usec, L: 0
0812 08:01:48.777058 (+   563us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:48.777113 (+    55us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:48.777121 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:48.777135 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:49.696878 (+919743us) raft_consensus.cc:1685] finished
0812 08:01:49.696881 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.696981 (+   100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:48.776855 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:48.776947 (+    92us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:48.776950 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:48.776952 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:48.777262 (+   310us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:48.777276 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:48.777282 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:48.777614 (+   332us) tablet.cc:701] Row locks acquired
0812 08:01:48.777616 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":273,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":826,"thread_start_us":287,"threads_started":1}]]}
I20260812 08:01:49.697623 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.34]   Dedup: 1.34->[]
W20260812 08:01:49.697700 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 173) took 869 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.697746 32085 rpcz_store.cc:269] 0812 08:01:48.828493 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.828637 (+   144us) service_pool.cc:224] Handling call
0812 08:01:49.696993 (+868356us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:49.697357 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 33
0812 08:01:49.697421 (+    64us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.697436 (+    15us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:49.697442 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.697459 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.697645 (+   186us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:49.697683 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1996553088}
I20260812 08:01:49.698051 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.34]   Dedup: 1.34->[]
W20260812 08:01:49.698081 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 175) took 410 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.698262 32088 rpcz_store.cc:269] 0812 08:01:49.287739 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.287857 (+   118us) service_pool.cc:224] Handling call
0812 08:01:49.697584 (+409727us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:49.697752 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 33
0812 08:01:49.697758 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.697766 (+     8us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:49.697769 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.697779 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.698038 (+   259us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:49.698068 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941741312}
I20260812 08:01:49.698369 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.34]   Dedup: 1.34->[]
W20260812 08:01:49.698369 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 174) took 462 ms (client timeout 50 ms). Trace:
W20260812 08:01:49.698565 32086 rpcz_store.cc:269] 0812 08:01:49.235756 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.235991 (+   235us) service_pool.cc:224] Handling call
0812 08:01:49.698019 (+462028us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:49.698166 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 33
0812 08:01:49.698172 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:49.698183 (+    11us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:49.698187 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.698197 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:49.698330 (+   133us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:49.698357 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1062099200}
I20260812 08:01:49.756629 32630 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=32, attempt_no=0}) took 22803 ms. Trace:
W20260812 08:01:49.757014 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1197ms of latency in SegmentAllocator::Sync()
I20260812 08:01:49.756806 32630 rpcz_store.cc:276] 0812 08:01:26.952924 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:26.953204 (+   280us) service_pool.cc:224] Handling call
0812 08:01:49.756606 (+22803402us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:47.634373 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:47.634495 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:47.634500 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:47.634502 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:47.634896 (+   394us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:47.634904 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:47.634907 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:48.775040 (+1140133us) lock_manager.cc:598] Waited 1139800us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:48.775062 (+    22us) tablet.cc:701] Row locks acquired
0812 08:01:48.775065 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:01:48.775116 (+    51us) write_op.cc:270] Start()
0812 08:01:48.775140 (+    24us) write_op.cc:276] Timestamp: P: 1786521708775064 usec, L: 0
0812 08:01:48.775142 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:48.775318 (+   176us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:49.754111 (+978793us) op_driver.cc:464] REPLICATION: finished
0812 08:01:49.754464 (+   353us) write_op.cc:301] APPLY: starting
0812 08:01:49.754503 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:49.754560 (+    57us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:49.754562 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:49.756194 (+  1632us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:49.756259 (+    65us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:49.756265 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:49.756400 (+   135us) log.cc:844] Serialized 319 byte log entry
0812 08:01:49.756478 (+    78us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:49.756515 (+    37us) write_op.cc:454] Released schema lock
0812 08:01:49.756524 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":272,"num_ops":50,"prepare.queue_time_us":20680325,"prepare.run_cpu_time_us":1482,"prepare.run_wall_time_us":1141069,"raft.queue_time_us":61,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":185,"replication_time_us":978922,"row_lock_wait_count":1,"row_lock_wait_us":1139800,"thread_start_us":116,"threads_started":1}]]}
W20260812 08:01:49.759122 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 963ms of latency in SegmentAllocator::Sync()
W20260812 08:01:49.859340 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:50.201637 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 200) took 952 ms (client timeout 50 ms). Trace:
I20260812 08:01:50.201663 31938 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.34->[1.35-1.35]
W20260812 08:01:50.201817 31937 rpcz_store.cc:269] 0812 08:01:49.249240 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.249400 (+   160us) service_pool.cc:224] Handling call
0812 08:01:49.249683 (+   283us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:49.249931 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 33
0812 08:01:49.250173 (+   242us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:49.250266 (+    93us) write_op.cc:270] Start()
0812 08:01:49.250292 (+    26us) write_op.cc:276] Timestamp: P: 1786521708775064 usec, L: 0
0812 08:01:49.250627 (+   335us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:49.251019 (+   392us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:49.251031 (+    12us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.251050 (+    19us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:50.201469 (+950419us) raft_consensus.cc:1685] finished
0812 08:01:50.201480 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.201601 (+   121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:49.250534 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:49.250777 (+   243us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:49.250782 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:49.250784 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:49.251412 (+   628us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:49.251425 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:49.251430 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:49.252472 (+  1042us) tablet.cc:701] Row locks acquired
0812 08:01:49.252474 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":1345,"prepare.run_wall_time_us":2038,"replication_time_us":951636,"row_lock_wait_count":1,"row_lock_wait_us":744,"spinlock_wait_cycles":77696,"thread_start_us":70,"threads_started":1}]]}
W20260812 08:01:50.202858 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 883ms of latency in SegmentAllocator::Sync()
W20260812 08:01:50.722899 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 178) took 965 ms (client timeout 50 ms). Trace:
I20260812 08:01:50.722954 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.35]   Dedup: 1.35->[]
W20260812 08:01:50.723042 32086 rpcz_store.cc:269] 0812 08:01:49.757606 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.757661 (+    55us) service_pool.cc:224] Handling call
0812 08:01:49.757863 (+   202us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:49.757970 (+   107us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:49.757974 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:49.758327 (+   353us) write_op.cc:270] Start()
0812 08:01:49.758358 (+    31us) write_op.cc:276] Timestamp: P: 1786521709756554 usec, L: 0
0812 08:01:49.758738 (+   380us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:49.758760 (+    22us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:49.758766 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:49.758780 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:50.722770 (+963990us) raft_consensus.cc:1685] finished
0812 08:01:50.722773 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.722872 (+    99us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:49.758939 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:49.759087 (+   148us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:49.759091 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:49.759093 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:49.759394 (+   301us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:49.759407 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:49.759411 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:49.760100 (+   689us) tablet.cc:701] Row locks acquired
0812 08:01:49.760103 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":459,"prepare.run_cpu_time_us":1238,"prepare.run_wall_time_us":1239,"thread_start_us":89,"threads_started":1}]]}
I20260812 08:01:50.723472 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.35]   Dedup: 1.35->[]
W20260812 08:01:50.723516 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 181) took 485 ms (client timeout 50 ms). Trace:
W20260812 08:01:50.723634 32084 rpcz_store.cc:269] 0812 08:01:50.238476 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.238746 (+   270us) service_pool.cc:224] Handling call
0812 08:01:50.722901 (+484155us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:50.723254 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:50.723256 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:50.723267 (+    11us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:50.723270 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.723281 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.723461 (+   180us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:50.723500 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112579840}
I20260812 08:01:50.723794 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.35]   Dedup: 1.35->[]
W20260812 08:01:50.723891 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 180) took 536 ms (client timeout 50 ms). Trace:
W20260812 08:01:50.723980 32088 rpcz_store.cc:269] 0812 08:01:50.187225 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.187446 (+   221us) service_pool.cc:224] Handling call
0812 08:01:50.723438 (+535992us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:50.723659 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:50.723663 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:50.723672 (+     9us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:50.723676 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.723695 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.723842 (+   147us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:50.723875 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1232053504}
I20260812 08:01:50.724072 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.35]   Dedup: 1.35->[]
W20260812 08:01:50.724126 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 179) took 915 ms (client timeout 50 ms). Trace:
W20260812 08:01:50.724196 32090 rpcz_store.cc:269] 0812 08:01:49.808813 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.808917 (+   104us) service_pool.cc:224] Handling call
0812 08:01:50.723762 (+914845us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:50.723940 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:50.723946 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:50.723956 (+    10us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:50.723962 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.723976 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.724090 (+   114us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:50.724114 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2103482752}
W20260812 08:01:50.724359 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 182) took 46 ms (client timeout 50 ms). Trace:
W20260812 08:01:50.724421 32085 rpcz_store.cc:269] 0812 08:01:50.678129 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.678222 (+    93us) service_pool.cc:224] Handling call
0812 08:01:50.724047 (+ 45825us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:50.724188 (+   141us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:50.724192 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:50.724202 (+    10us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:50.724206 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.724218 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:50.724321 (+   103us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:50.724347 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":104927104}
W20260812 08:01:50.757916 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1340
W20260812 08:01:50.935675 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 16: this message will repeat every 5th retry.
I20260812 08:01:50.957401 32638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=33, attempt_no=0}) took 22393 ms. Trace:
I20260812 08:01:50.957630 32638 rpcz_store.cc:276] 0812 08:01:28.563999 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:28.564086 (+    87us) service_pool.cc:224] Handling call
0812 08:01:50.957376 (+22393290us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:48.775476 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:48.775597 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:48.775602 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:48.775603 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:48.776021 (+   418us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:48.776029 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:48.776033 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:49.756524 (+980491us) lock_manager.cc:598] Waited 980193us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:49.756537 (+    13us) tablet.cc:701] Row locks acquired
0812 08:01:49.756539 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:49.756635 (+    96us) write_op.cc:270] Start()
0812 08:01:49.756663 (+    28us) write_op.cc:276] Timestamp: P: 1786521709756554 usec, L: 0
0812 08:01:49.756667 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:49.756832 (+   165us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:50.955049 (+1198217us) op_driver.cc:464] REPLICATION: finished
0812 08:01:50.955399 (+   350us) write_op.cc:301] APPLY: starting
0812 08:01:50.955438 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:50.955494 (+    56us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:50.955495 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:50.956919 (+  1424us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:50.956967 (+    48us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:50.956972 (+     5us) write_op.cc:312] APPLY: finished
0812 08:01:50.957172 (+   200us) log.cc:844] Serialized 319 byte log entry
0812 08:01:50.957247 (+    75us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:50.957279 (+    32us) write_op.cc:454] Released schema lock
0812 08:01:50.957288 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":259,"num_ops":50,"prepare.queue_time_us":20211052,"prepare.run_cpu_time_us":1287,"prepare.run_wall_time_us":981454,"raft.queue_time_us":567,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":244,"replication_time_us":1198331,"row_lock_wait_count":1,"row_lock_wait_us":980193,"spinlock_wait_cycles":87296,"thread_start_us":108,"threads_started":1}]]}
W20260812 08:01:50.958412 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 894ms of latency in SegmentAllocator::Sync()
W20260812 08:01:50.961012 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1144ms of latency in SegmentAllocator::Sync()
W20260812 08:01:51.061544 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:51.086474 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.34]   Dedup: 1.34->[]
W20260812 08:01:51.086745 31938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 202) took 1235 ms (client timeout 50 ms). Trace:
I20260812 08:01:51.086863 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.35->[]
W20260812 08:01:51.086879 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 201) took 1786 ms (client timeout 50 ms). Trace:
W20260812 08:01:51.087100 31939 rpcz_store.cc:269] 0812 08:01:49.300273 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.300416 (+   143us) service_pool.cc:224] Handling call
0812 08:01:51.086421 (+1786005us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:01:51.086682 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 33
0812 08:01:51.086686 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.086698 (+    12us) raft_consensus.cc:1645] Marking committed up to 33
0812 08:01:51.086702 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.086714 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.086841 (+   127us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.086869 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4106481536}
W20260812 08:01:51.086848 31938 rpcz_store.cc:269] 0812 08:01:49.851176 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.851337 (+   161us) service_pool.cc:224] Handling call
0812 08:01:50.201575 (+350238us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:50.201879 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:50.202093 (+   214us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:50.202258 (+   165us) write_op.cc:270] Start()
0812 08:01:50.202294 (+    36us) write_op.cc:276] Timestamp: P: 1786521709756554 usec, L: 0
0812 08:01:50.202666 (+   372us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:50.203022 (+   356us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:50.203030 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.203046 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:51.086308 (+883262us) raft_consensus.cc:1685] finished
0812 08:01:51.086310 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.086562 (+   252us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.086653 (+    91us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:50.202553 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:50.202700 (+   147us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:50.202705 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:50.202706 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:50.203213 (+   507us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:50.203229 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:50.203232 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:50.204558 (+  1326us) tablet.cc:701] Row locks acquired
0812 08:01:50.204560 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":804888832,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":1031,"prepare.run_wall_time_us":2058,"row_lock_wait_count":1,"row_lock_wait_us":1100,"thread_start_us":86,"threads_started":1}]]}
I20260812 08:01:51.087383 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.35->[]
W20260812 08:01:51.087431 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 206) took 252 ms (client timeout 50 ms). Trace:
W20260812 08:01:51.087478 31942 rpcz_store.cc:269] 0812 08:01:50.835062 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.835183 (+   121us) service_pool.cc:224] Handling call
0812 08:01:51.086797 (+251614us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:51.087031 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:51.087037 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.087048 (+    11us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:51.087052 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.087063 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.087378 (+   315us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.087417 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":578003328}
W20260812 08:01:51.087756 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 205) took 659 ms (client timeout 50 ms). Trace:
I20260812 08:01:51.087806 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.35->[]
W20260812 08:01:51.087821 31941 rpcz_store.cc:269] 0812 08:01:50.428170 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.428278 (+   108us) service_pool.cc:224] Handling call
0812 08:01:51.087260 (+658982us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:51.087503 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:51.087506 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.087513 (+     7us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:51.087517 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.087541 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.087701 (+   160us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.087743 (+    42us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1514576128}
I20260812 08:01:51.088089 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.35->[]
W20260812 08:01:51.088167 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 204) took 711 ms (client timeout 50 ms). Trace:
W20260812 08:01:51.088214 31937 rpcz_store.cc:269] 0812 08:01:50.377066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.377251 (+   185us) service_pool.cc:224] Handling call
0812 08:01:51.087767 (+710516us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:51.087947 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:51.087951 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.087963 (+    12us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:51.087968 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.087978 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.088123 (+   145us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.088152 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633220864}
W20260812 08:01:51.088388 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 203) took 1185 ms (client timeout 50 ms). Trace:
W20260812 08:01:51.088436 31936 rpcz_store.cc:269] 0812 08:01:49.902539 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.902698 (+   159us) service_pool.cc:224] Handling call
0812 08:01:51.088055 (+1185357us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:51.088222 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:51.088227 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.088235 (+     8us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:51.088239 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.088250 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.088350 (+   100us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.088377 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2725035776}
I20260812 08:01:51.088533 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.35]   Dedup: 1.35->[]
W20260812 08:01:51.088876 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 207) took 203 ms (client timeout 50 ms). Trace:
W20260812 08:01:51.088934 31935 rpcz_store.cc:269] 0812 08:01:50.885939 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.886072 (+   133us) service_pool.cc:224] Handling call
0812 08:01:51.088504 (+202432us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:01:51.088635 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 34
0812 08:01:51.088641 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:51.088651 (+    10us) raft_consensus.cc:1645] Marking committed up to 34
0812 08:01:51.088728 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.088744 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:51.088843 (+    99us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:51.088865 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464714624}
I20260812 08:01:51.257786 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.35->[1.36-1.36]
W20260812 08:01:51.259282 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1177ms of latency in SegmentAllocator::Sync()
W20260812 08:01:51.959124 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9b80
W20260812 08:01:52.105786 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 184) took 1145 ms (client timeout 50 ms). Trace:
I20260812 08:01:52.105870 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.106081 32085 rpcz_store.cc:269] 0812 08:01:50.959894 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.959996 (+   102us) service_pool.cc:224] Handling call
0812 08:01:50.960226 (+   230us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:50.960296 (+    70us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:50.960298 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:50.960370 (+    72us) write_op.cc:270] Start()
0812 08:01:50.960393 (+    23us) write_op.cc:276] Timestamp: P: 1786521710957655 usec, L: 0
0812 08:01:50.960665 (+   272us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:50.960845 (+   180us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:50.960852 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:50.960866 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:52.105605 (+1144739us) raft_consensus.cc:1685] finished
0812 08:01:52.105608 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.105752 (+   144us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:50.960670 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:50.960796 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:50.960801 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:50.960803 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:50.961487 (+   684us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:50.961506 (+    19us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:50.961510 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:50.961967 (+   457us) tablet.cc:701] Row locks acquired
0812 08:01:50.961969 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":201,"prepare.run_cpu_time_us":1360,"prepare.run_wall_time_us":1359,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:01:52.106508 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.106642 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 186) took 550 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.106700 32088 rpcz_store.cc:269] 0812 08:01:51.555885 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.555992 (+   107us) service_pool.cc:224] Handling call
0812 08:01:52.105819 (+549827us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.106252 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.106256 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.106347 (+    91us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.106355 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.106376 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.106567 (+   191us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:52.106624 (+    57us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1263994624}
I20260812 08:01:52.106805 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.106863 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 188) took 109 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.106909 32086 rpcz_store.cc:269] 0812 08:01:51.996986 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.997109 (+   123us) service_pool.cc:224] Handling call
0812 08:01:52.106470 (+109361us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.106678 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.106681 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.106689 (+     8us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.106697 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.106719 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.106829 (+   110us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:52.106852 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":251101056}
I20260812 08:01:52.107048 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.107124 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 185) took 1095 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.107184 32090 rpcz_store.cc:269] 0812 08:01:51.011538 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.011656 (+   118us) service_pool.cc:224] Handling call
0812 08:01:52.106775 (+1095119us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.106942 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.106947 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.106955 (+     8us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.106959 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.106970 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.107088 (+   118us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:52.107112 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2518083328}
I20260812 08:01:52.107301 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.107342 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 189) took 59 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.107388 32089 rpcz_store.cc:269] 0812 08:01:52.048057 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.048203 (+   146us) service_pool.cc:224] Handling call
0812 08:01:52.107022 (+ 58819us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.107143 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.107148 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.107158 (+    10us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.107164 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.107177 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.107302 (+   125us) spinlock_profiling.cc:243] Waited 58.6 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:52.107330 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134824832}
W20260812 08:01:52.107630 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 187) took 500 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.107726 32084 rpcz_store.cc:269] 0812 08:01:51.607185 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.607311 (+   126us) service_pool.cc:224] Handling call
0812 08:01:52.107274 (+499963us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.107457 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.107463 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.107471 (+     8us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.107476 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.107489 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.107593 (+   104us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:52.107619 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1149269248}
W20260812 08:01:52.436977 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 208) took 1180 ms (client timeout 50 ms). Trace:
I20260812 08:01:52.436977 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.437326 31935 rpcz_store.cc:269] 0812 08:01:51.256899 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.257019 (+   120us) service_pool.cc:224] Handling call
0812 08:01:51.257428 (+   409us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:51.258052 (+   624us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:51.258261 (+   209us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:51.258369 (+   108us) write_op.cc:270] Start()
0812 08:01:51.258394 (+    25us) write_op.cc:276] Timestamp: P: 1786521710957655 usec, L: 0
0812 08:01:51.258861 (+   467us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:51.258906 (+    45us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:51.258912 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:51.258925 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:52.436805 (+1177880us) raft_consensus.cc:1685] finished
0812 08:01:52.436808 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.436947 (+   139us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:51.258739 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:51.258861 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:51.258866 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:51.258867 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:51.259163 (+   296us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:51.259172 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:51.259174 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:51.260638 (+  1464us) tablet.cc:701] Row locks acquired
0812 08:01:51.260640 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":265,"prepare.run_cpu_time_us":789,"prepare.run_wall_time_us":1960,"row_lock_wait_count":1,"row_lock_wait_us":1258,"thread_start_us":187,"threads_started":1}]]}
W20260812 08:01:52.437590 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 209) took 1129 ms (client timeout 50 ms). Trace:
I20260812 08:01:52.437538 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.437646 31936 rpcz_store.cc:269] 0812 08:01:51.307991 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.308103 (+   112us) service_pool.cc:224] Handling call
0812 08:01:52.436919 (+1128816us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:52.437397 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.437401 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.437414 (+    13us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.437419 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.437433 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.437548 (+   115us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:52.437578 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595359104}
I20260812 08:01:52.437897 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.437927 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 210) took 620 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.438491 31937 rpcz_store.cc:269] 0812 08:01:51.817213 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.817341 (+   128us) service_pool.cc:224] Handling call
0812 08:01:52.437504 (+620163us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:52.437710 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.437714 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.437724 (+    10us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.437731 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.437747 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.437881 (+   134us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:52.437913 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1425609984}
I20260812 08:01:52.438673 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.438721 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 211) took 570 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.438773 31941 rpcz_store.cc:269] 0812 08:01:51.868021 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:51.868137 (+   116us) service_pool.cc:224] Handling call
0812 08:01:52.437858 (+569721us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:52.438505 (+   647us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.438512 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.438522 (+    10us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.438525 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.438536 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.438687 (+   151us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:52.438711 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1309615872}
W20260812 08:01:52.439158 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 213) took 172 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.439239 31939 rpcz_store.cc:269] 0812 08:01:52.266992 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.267126 (+   134us) service_pool.cc:224] Handling call
0812 08:01:52.438636 (+171510us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:52.438779 (+   143us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.438782 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.438789 (+     7us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.438793 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.438803 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.439103 (+   300us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:52.439137 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393773440}
I20260812 08:01:52.439576 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.36]   Dedup: 1.36->[]
W20260812 08:01:52.439878 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 212) took 224 ms (client timeout 50 ms). Trace:
W20260812 08:01:52.439957 31942 rpcz_store.cc:269] 0812 08:01:52.215414 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.215568 (+   154us) service_pool.cc:224] Handling call
0812 08:01:52.439538 (+223970us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:01:52.439690 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 35
0812 08:01:52.439695 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:52.439706 (+    11us) raft_consensus.cc:1645] Marking committed up to 35
0812 08:01:52.439710 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.439721 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:52.439837 (+   116us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:52.439866 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514490752}
I20260812 08:01:52.602006 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.36]   Dedup: 1.36->[]
I20260812 08:01:52.608842 32648 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=34, attempt_no=0}) took 22940 ms. Trace:
I20260812 08:01:52.608984 32648 rpcz_store.cc:276] 0812 08:01:29.668259 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:29.668374 (+   115us) service_pool.cc:224] Handling call
0812 08:01:52.608821 (+22940447us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:49.756958 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:49.757077 (+   119us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:49.757081 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:49.757083 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:49.757412 (+   329us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:49.757421 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:49.757424 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:50.957659 (+1200235us) lock_manager.cc:598] Waited 1199923us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:50.957674 (+    15us) tablet.cc:701] Row locks acquired
0812 08:01:50.957676 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:50.957725 (+    49us) write_op.cc:270] Start()
0812 08:01:50.957751 (+    26us) write_op.cc:276] Timestamp: P: 1786521710957655 usec, L: 0
0812 08:01:50.957766 (+    15us) op_driver.cc:348] REPLICATION: starting
0812 08:01:50.958155 (+   389us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:52.603264 (+1645109us) op_driver.cc:464] REPLICATION: finished
0812 08:01:52.603493 (+   229us) write_op.cc:301] APPLY: starting
0812 08:01:52.603556 (+    63us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:52.603645 (+    89us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:52.603649 (+     4us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:52.608379 (+  4730us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:52.608428 (+    49us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:52.608432 (+     4us) write_op.cc:312] APPLY: finished
0812 08:01:52.608569 (+   137us) log.cc:844] Serialized 319 byte log entry
0812 08:01:52.608712 (+   143us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:52.608738 (+    26us) write_op.cc:454] Released schema lock
0812 08:01:52.608745 (+     7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":148,"num_ops":50,"prepare.queue_time_us":20088182,"prepare.run_cpu_time_us":1487,"prepare.run_wall_time_us":1201289,"raft.queue_time_us":1639,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":214,"replication_time_us":1645458,"row_lock_wait_count":1,"row_lock_wait_us":1199923,"spinlock_wait_cycles":504576,"thread_start_us":102,"threads_started":1}]]}
W20260812 08:01:52.609431 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 920ms of latency in SegmentAllocator::Sync()
W20260812 08:01:52.611160 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1201ms of latency in SegmentAllocator::Sync()
W20260812 08:01:52.711866 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:52.823230 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.36->[1.37-1.37]
W20260812 08:01:52.824493 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 910ms of latency in SegmentAllocator::Sync()
W20260812 08:01:53.312472 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 21: this message will repeat every 5th retry.
W20260812 08:01:53.610321 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8000
W20260812 08:01:53.735093 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 214) took 912 ms (client timeout 50 ms). Trace:
I20260812 08:01:53.735176 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.37->[]
I20260812 08:01:53.735536 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.735200 31942 rpcz_store.cc:269] 0812 08:01:52.822439 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.822642 (+   203us) service_pool.cc:224] Handling call
0812 08:01:52.823158 (+   516us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:52.823383 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:52.823651 (+   268us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:52.823817 (+   166us) write_op.cc:270] Start()
0812 08:01:52.823844 (+    27us) write_op.cc:276] Timestamp: P: 1786521712608810 usec, L: 0
0812 08:01:52.824213 (+   369us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:52.824254 (+    41us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:52.824260 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.824276 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:53.734952 (+910676us) raft_consensus.cc:1685] finished
0812 08:01:53.734955 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.735060 (+   105us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:52.824137 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:52.824264 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:52.824269 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:52.824271 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:52.824626 (+   355us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:52.824637 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:52.824641 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:52.825568 (+   927us) tablet.cc:701] Row locks acquired
0812 08:01:52.825570 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":187,"prepare.run_cpu_time_us":917,"prepare.run_wall_time_us":1532,"row_lock_wait_count":1,"row_lock_wait_us":651,"spinlock_wait_cycles":7936,"thread_start_us":93,"threads_started":1}]]}
W20260812 08:01:53.735615 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 216) took 524 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.735702 31941 rpcz_store.cc:269] 0812 08:01:53.211315 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.211426 (+   111us) service_pool.cc:224] Handling call
0812 08:01:53.735063 (+523637us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:53.735397 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.735400 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.735412 (+    12us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.735416 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.735428 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.735568 (+   140us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:53.735599 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203279744}
I20260812 08:01:53.735899 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.37->[]
I20260812 08:01:53.736167 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.736243 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 217) took 473 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.736289 31937 rpcz_store.cc:269] 0812 08:01:53.262364 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.262503 (+   139us) service_pool.cc:224] Handling call
0812 08:01:53.735869 (+473366us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:53.736008 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.736014 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.736025 (+    11us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.736033 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.736048 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.736196 (+   148us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:53.736227 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087779712}
W20260812 08:01:53.736430 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 215) took 862 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.736477 31939 rpcz_store.cc:269] 0812 08:01:52.873524 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.873626 (+   102us) service_pool.cc:224] Handling call
0812 08:01:53.736131 (+862505us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:01:53.736274 (+   143us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.736277 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.736286 (+     9us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.736290 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.736302 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.736397 (+    95us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:53.736418 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982766976}
I20260812 08:01:53.738999 32658 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=35, attempt_no=0}) took 22976 ms. Trace:
I20260812 08:01:53.739099 32658 rpcz_store.cc:276] 0812 08:01:30.762446 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:30.762522 (+    76us) service_pool.cc:224] Handling call
0812 08:01:53.738979 (+22976457us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:50.958273 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:50.958390 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:50.958394 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:50.958396 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:50.958698 (+   302us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:50.958707 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:50.958710 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:52.608836 (+1650126us) lock_manager.cc:598] Waited 1649835us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:52.608862 (+    26us) tablet.cc:701] Row locks acquired
0812 08:01:52.608865 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:01:52.608917 (+    52us) write_op.cc:270] Start()
0812 08:01:52.608944 (+    27us) write_op.cc:276] Timestamp: P: 1786521712608810 usec, L: 0
0812 08:01:52.608946 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:52.609148 (+   202us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:53.736688 (+1127540us) op_driver.cc:464] REPLICATION: finished
0812 08:01:53.737003 (+   315us) write_op.cc:301] APPLY: starting
0812 08:01:53.737047 (+    44us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:53.737113 (+    66us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:53.737114 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:53.738518 (+  1404us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:53.738573 (+    55us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:53.738628 (+    55us) write_op.cc:312] APPLY: finished
0812 08:01:53.738777 (+   149us) log.cc:844] Serialized 319 byte log entry
0812 08:01:53.738853 (+    76us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:53.738880 (+    27us) write_op.cc:454] Released schema lock
0812 08:01:53.738896 (+    16us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"num_ops":50,"prepare.queue_time_us":20195294,"prepare.run_cpu_time_us":1452,"prepare.run_wall_time_us":1650990,"raft.queue_time_us":574,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":452,"replication_time_us":1127703,"row_lock_wait_count":1,"row_lock_wait_us":1649835,"spinlock_wait_cycles":2816,"thread_start_us":147,"threads_started":1}]]}
W20260812 08:01:53.739946 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1038ms of latency in SegmentAllocator::Sync()
W20260812 08:01:53.741269 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 922ms of latency in SegmentAllocator::Sync()
W20260812 08:01:53.812743 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 192) took 1202 ms (client timeout 50 ms). Trace:
I20260812 08:01:53.812793 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.812906 32084 rpcz_store.cc:269] 0812 08:01:52.610073 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.610152 (+    79us) service_pool.cc:224] Handling call
0812 08:01:52.610330 (+   178us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:52.610417 (+    87us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:52.610420 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:52.610556 (+   136us) write_op.cc:270] Start()
0812 08:01:52.610611 (+    55us) write_op.cc:276] Timestamp: P: 1786521712608810 usec, L: 0
0812 08:01:52.610935 (+   324us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:52.610993 (+    58us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:52.611007 (+    14us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:52.611020 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:53.812603 (+1201583us) raft_consensus.cc:1685] finished
0812 08:01:53.812607 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.812709 (+   102us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:52.610895 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:52.611080 (+   185us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:52.611085 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:52.611087 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:52.611563 (+   476us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:52.611576 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:52.611580 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:52.611979 (+   399us) tablet.cc:701] Row locks acquired
0812 08:01:52.611981 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":1173,"thread_start_us":86,"threads_started":1}]]}
I20260812 08:01:53.813292 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.813330 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 193) took 1151 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.813377 32089 rpcz_store.cc:269] 0812 08:01:52.661789 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.661934 (+   145us) service_pool.cc:224] Handling call
0812 08:01:53.812738 (+1150804us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.813112 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.813115 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.813125 (+    10us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.813128 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.813138 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.813263 (+   125us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:53.813302 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2646103296}
W20260812 08:01:53.813618 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 194) took 595 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.813678 32090 rpcz_store.cc:269] 0812 08:01:53.218292 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.218405 (+   113us) service_pool.cc:224] Handling call
0812 08:01:53.813263 (+594858us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.813401 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.813404 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.813411 (+     7us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.813415 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.813425 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.813575 (+   150us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:53.813602 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1367534848}
I20260812 08:01:53.813856 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.814172 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 195) took 544 ms (client timeout 50 ms). Trace:
I20260812 08:01:53.814093 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.814244 32086 rpcz_store.cc:269] 0812 08:01:53.269354 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.269470 (+   116us) service_pool.cc:224] Handling call
0812 08:01:53.813792 (+544322us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.813985 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.813991 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.814002 (+    11us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.814009 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.814019 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.814132 (+   113us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:53.814159 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1251258624}
I20260812 08:01:53.814457 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37]   Dedup: 1.37->[]
W20260812 08:01:53.814457 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 196) took 172 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.814545 32088 rpcz_store.cc:269] 0812 08:01:53.641908 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.642008 (+   100us) service_pool.cc:224] Handling call
0812 08:01:53.814065 (+172057us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.814296 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.814299 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.814306 (+     7us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.814309 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.814319 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.814423 (+   104us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:53.814447 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395187456}
W20260812 08:01:53.814750 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 197) took 121 ms (client timeout 50 ms). Trace:
W20260812 08:01:53.814838 32085 rpcz_store.cc:269] 0812 08:01:53.693263 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.693371 (+   108us) service_pool.cc:224] Handling call
0812 08:01:53.814439 (+121068us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.814569 (+   130us) raft_consensus.cc:1534] Early marking committed up to index 36
0812 08:01:53.814571 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:53.814599 (+    28us) raft_consensus.cc:1645] Marking committed up to 36
0812 08:01:53.814604 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.814613 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:53.814716 (+   103us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:53.814740 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277904768}
W20260812 08:01:53.841522 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:54.224431 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.38]   Dedup: 1.37->[1.38-1.38]
W20260812 08:01:54.225644 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1000ms of latency in SegmentAllocator::Sync()
W20260812 08:01:54.274053 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 20 similar messages]
W20260812 08:01:54.663843 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 220) took 923 ms (client timeout 50 ms). Trace:
I20260812 08:01:54.663976 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.38]   Dedup: 1.38->[]
W20260812 08:01:54.664088 31939 rpcz_store.cc:269] 0812 08:01:53.740335 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.740388 (+    53us) service_pool.cc:224] Handling call
0812 08:01:53.740526 (+   138us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:53.740601 (+    75us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:53.740604 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:53.740709 (+   105us) write_op.cc:270] Start()
0812 08:01:53.740730 (+    21us) write_op.cc:276] Timestamp: P: 1786521713738888 usec, L: 0
0812 08:01:53.741033 (+   303us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:53.741073 (+    40us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:53.741078 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:53.741091 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:54.663684 (+922593us) raft_consensus.cc:1685] finished
0812 08:01:54.663686 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:54.663804 (+   118us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:53.740981 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:53.741104 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:53.741111 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:01:53.741114 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:53.741421 (+   307us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:53.741433 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:53.741436 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:53.741622 (+   186us) tablet.cc:701] Row locks acquired
0812 08:01:53.741623 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":701,"prepare.run_wall_time_us":700,"thread_start_us":99,"threads_started":1}]]}
I20260812 08:01:54.664495 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.38]   Dedup: 1.38->[]
W20260812 08:01:54.664654 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 221) took 873 ms (client timeout 50 ms). Trace:
W20260812 08:01:54.664722 31937 rpcz_store.cc:269] 0812 08:01:53.791454 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.791531 (+    77us) service_pool.cc:224] Handling call
0812 08:01:54.663857 (+872326us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:54.664294 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 37
0812 08:01:54.664297 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:54.664326 (+    29us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:54.664329 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:54.664341 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:54.664514 (+   173us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:54.664636 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005614464}
I20260812 08:01:54.664841 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.38]   Dedup: 1.38->[]
W20260812 08:01:54.664880 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 222) took 339 ms (client timeout 50 ms). Trace:
W20260812 08:01:54.664934 31941 rpcz_store.cc:269] 0812 08:01:54.325379 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.325444 (+    65us) service_pool.cc:224] Handling call
0812 08:01:54.664450 (+339006us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:54.664683 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 37
0812 08:01:54.664687 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:54.664695 (+     8us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:54.664700 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:54.664713 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:54.664837 (+   124us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:54.664860 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":778927488}
W20260812 08:01:54.665153 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 223) took 288 ms (client timeout 50 ms). Trace:
W20260812 08:01:54.665222 31936 rpcz_store.cc:269] 0812 08:01:54.376337 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.376465 (+   128us) service_pool.cc:224] Handling call
0812 08:01:54.664815 (+288350us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:54.664964 (+   149us) raft_consensus.cc:1534] Early marking committed up to index 37
0812 08:01:54.664970 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:54.664982 (+    12us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:54.664987 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:54.664999 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:54.665113 (+   114us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:54.665141 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662676352}
I20260812 08:01:54.732002 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.38]   Dedup: 1.38->[]
W20260812 08:01:54.740324 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9080
I20260812 08:01:54.782114 32664 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=36, attempt_no=0}) took 22961 ms. Trace:
W20260812 08:01:54.782459 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1187ms of latency in SegmentAllocator::Sync()
I20260812 08:01:54.782387 32664 rpcz_store.cc:276] 0812 08:01:31.821017 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:31.821144 (+   127us) service_pool.cc:224] Handling call
0812 08:01:54.782091 (+22960947us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:52.609298 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:52.609465 (+   167us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:52.609470 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:52.609472 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:52.609939 (+   467us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:52.609949 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:52.609953 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:53.738909 (+1128956us) lock_manager.cc:598] Waited 1128711us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:53.738921 (+    12us) tablet.cc:701] Row locks acquired
0812 08:01:53.738923 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:53.738996 (+    73us) write_op.cc:270] Start()
0812 08:01:53.739023 (+    27us) write_op.cc:276] Timestamp: P: 1786521713738888 usec, L: 0
0812 08:01:53.739027 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:01:53.739219 (+   192us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:54.778625 (+1039406us) op_driver.cc:464] REPLICATION: finished
0812 08:01:54.779033 (+   408us) write_op.cc:301] APPLY: starting
0812 08:01:54.779076 (+    43us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:54.779145 (+    69us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:54.779146 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:54.781346 (+  2200us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:54.781411 (+    65us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:54.781417 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:54.781555 (+   138us) log.cc:844] Serialized 319 byte log entry
0812 08:01:54.781762 (+   207us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:54.781967 (+   205us) write_op.cc:454] Released schema lock
0812 08:01:54.781982 (+    15us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":335,"num_ops":50,"prepare.queue_time_us":20787671,"prepare.run_cpu_time_us":1593,"prepare.run_wall_time_us":1130018,"raft.queue_time_us":34,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":181,"replication_time_us":1039518,"row_lock_wait_count":1,"row_lock_wait_us":1128711,"spinlock_wait_cycles":88064,"thread_start_us":116,"threads_started":1}]]}
W20260812 08:01:54.784271 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1062ms of latency in SegmentAllocator::Sync()
W20260812 08:01:54.884832 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:55.226429 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 198) took 1002 ms (client timeout 50 ms). Trace:
I20260812 08:01:55.226552 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.38->[1.39-1.39]
W20260812 08:01:55.226691 32085 rpcz_store.cc:269] 0812 08:01:54.223987 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.224115 (+   128us) service_pool.cc:224] Handling call
0812 08:01:54.224326 (+   211us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:54.224589 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 37
0812 08:01:54.224847 (+   258us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:54.224938 (+    91us) write_op.cc:270] Start()
0812 08:01:54.224966 (+    28us) write_op.cc:276] Timestamp: P: 1786521713738888 usec, L: 0
0812 08:01:54.225368 (+   402us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:54.225413 (+    45us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:54.225419 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:54.225434 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:55.226137 (+1000703us) raft_consensus.cc:1685] finished
0812 08:01:55.226140 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.226395 (+   255us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:54.225265 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:54.225398 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:54.225403 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:54.225405 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:54.225815 (+   410us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:54.225836 (+    21us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:54.225841 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:54.227366 (+  1525us) tablet.cc:701] Row locks acquired
0812 08:01:54.227369 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":1047,"prepare.run_wall_time_us":2171,"row_lock_wait_count":1,"row_lock_wait_us":1272,"thread_start_us":96,"threads_started":1}]]}
W20260812 08:01:55.227763 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 957ms of latency in SegmentAllocator::Sync()
W20260812 08:01:55.451532 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:01:55.783528 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18dc0
I20260812 08:01:55.846939 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39]   Dedup: 1.39->[]
W20260812 08:01:55.847005 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 226) took 1063 ms (client timeout 50 ms). Trace:
I20260812 08:01:55.847472 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39]   Dedup: 1.39->[]
W20260812 08:01:55.847286 31936 rpcz_store.cc:269] 0812 08:01:54.783129 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.783253 (+   124us) service_pool.cc:224] Handling call
0812 08:01:54.783421 (+   168us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:54.783511 (+    90us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:54.783513 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:54.783632 (+   119us) write_op.cc:270] Start()
0812 08:01:54.783657 (+    25us) write_op.cc:276] Timestamp: P: 1786521714781981 usec, L: 0
0812 08:01:54.784031 (+   374us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:54.784081 (+    50us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:54.784087 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:54.784100 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:55.846830 (+1062730us) raft_consensus.cc:1685] finished
0812 08:01:55.846833 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.846968 (+   135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:54.784111 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:54.784233 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:54.784236 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:01:54.784239 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:54.784705 (+   466us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:54.784715 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:54.784718 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:54.784896 (+   178us) tablet.cc:701] Row locks acquired
0812 08:01:54.784897 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":850,"thread_start_us":115,"threads_started":1}]]}
W20260812 08:01:55.847841 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 230) took 62 ms (client timeout 50 ms). Trace:
I20260812 08:01:55.847935 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39]   Dedup: 1.39->[]
W20260812 08:01:55.847908 31942 rpcz_store.cc:269] 0812 08:01:55.785800 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.785898 (+    98us) service_pool.cc:224] Handling call
0812 08:01:55.847426 (+ 61528us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:55.847619 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:55.847625 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:55.847637 (+    12us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:55.847642 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.847652 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.847794 (+   142us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:55.847826 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140879744}
I20260812 08:01:55.848248 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39]   Dedup: 1.39->[]
W20260812 08:01:55.848338 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 227) took 1013 ms (client timeout 50 ms). Trace:
W20260812 08:01:55.848383 31941 rpcz_store.cc:269] 0812 08:01:54.834837 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.835026 (+   189us) service_pool.cc:224] Handling call
0812 08:01:55.847788 (+1012762us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:55.848110 (+   322us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:55.848115 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:55.848132 (+    17us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:55.848137 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.848148 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.848290 (+   142us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:55.848324 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2328633728}
I20260812 08:01:55.848497 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.39]   Dedup: 1.39->[]
W20260812 08:01:55.848516 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 228) took 655 ms (client timeout 50 ms). Trace:
W20260812 08:01:55.848655 31937 rpcz_store.cc:269] 0812 08:01:55.193549 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.193767 (+   218us) service_pool.cc:224] Handling call
0812 08:01:55.848216 (+654449us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:55.848354 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:55.848358 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:55.848366 (+     8us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:55.848370 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.848381 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.848485 (+   104us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:55.848505 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1504645376}
W20260812 08:01:55.848784 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 229) took 603 ms (client timeout 50 ms). Trace:
W20260812 08:01:55.848856 31939 rpcz_store.cc:269] 0812 08:01:55.245131 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.245270 (+   139us) service_pool.cc:224] Handling call
0812 08:01:55.848471 (+603201us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:55.848622 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:55.848625 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:55.848633 (+     8us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:55.848636 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.848645 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:55.848753 (+   108us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:55.848774 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1386796800}
I20260812 08:01:55.972784 32673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=37, attempt_no=0}) took 23115 ms. Trace:
W20260812 08:01:55.973169 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 819ms of latency in SegmentAllocator::Sync()
I20260812 08:01:55.973003 32673 rpcz_store.cc:276] 0812 08:01:32.857011 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:32.857106 (+    95us) service_pool.cc:224] Handling call
0812 08:01:55.972731 (+23115625us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:53.739348 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:53.739473 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:53.739479 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:53.739481 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:53.739812 (+   331us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:53.739820 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:53.739823 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:54.781979 (+1042156us) lock_manager.cc:598] Waited 1041782us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:54.781994 (+    15us) tablet.cc:701] Row locks acquired
0812 08:01:54.781996 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:54.782048 (+    52us) write_op.cc:270] Start()
0812 08:01:54.782076 (+    28us) write_op.cc:276] Timestamp: P: 1786521714781981 usec, L: 0
0812 08:01:54.782077 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:01:54.782251 (+   174us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:55.970208 (+1187957us) op_driver.cc:464] REPLICATION: finished
0812 08:01:55.970561 (+   353us) write_op.cc:301] APPLY: starting
0812 08:01:55.970661 (+   100us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:55.970727 (+    66us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:55.970729 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:55.972130 (+  1401us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:55.972193 (+    63us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:55.972199 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:55.972345 (+   146us) log.cc:844] Serialized 319 byte log entry
0812 08:01:55.972601 (+   256us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:55.972638 (+    37us) write_op.cc:454] Released schema lock
0812 08:01:55.972648 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":279,"num_ops":50,"prepare.queue_time_us":20881909,"prepare.run_cpu_time_us":1541,"prepare.run_wall_time_us":1043008,"raft.queue_time_us":530,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"replication_time_us":1188052,"row_lock_wait_count":1,"row_lock_wait_us":1041782,"thread_start_us":120,"threads_started":1}]]}
W20260812 08:01:55.975078 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1124ms of latency in SegmentAllocator::Sync()
W20260812 08:01:56.075244 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:01:56.185397 32088 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.38]   Dedup: 1.38->[]
W20260812 08:01:56.185550 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 200) took 1383 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.185758 32088 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 199) took 1910 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.185849 32088 rpcz_store.cc:269] 0812 08:01:54.275066 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.275204 (+   138us) service_pool.cc:224] Handling call
0812 08:01:56.185339 (+1910135us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:56.185575 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 37
0812 08:01:56.185579 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.185590 (+    11us) raft_consensus.cc:1645] Marking committed up to 37
0812 08:01:56.185595 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.185609 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.185725 (+   116us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.185748 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4392394112}
I20260812 08:01:56.185762 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.39->[]
W20260812 08:01:56.185606 32086 rpcz_store.cc:269] 0812 08:01:54.802414 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.802566 (+   152us) service_pool.cc:224] Handling call
0812 08:01:55.226370 (+423804us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:55.226837 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:55.227047 (+   210us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:55.227145 (+    98us) write_op.cc:270] Start()
0812 08:01:55.227179 (+    34us) write_op.cc:276] Timestamp: P: 1786521714781981 usec, L: 0
0812 08:01:55.227493 (+   314us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:55.227536 (+    43us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:55.227541 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.227557 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:56.185218 (+957661us) raft_consensus.cc:1685] finished
0812 08:01:56.185221 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.185464 (+   243us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.185529 (+    65us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:55.227415 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:55.227538 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:55.227542 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:55.227543 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:55.227949 (+   406us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:55.227961 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:55.227964 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:55.229026 (+  1062us) tablet.cc:701] Row locks acquired
0812 08:01:55.229028 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":973943552,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":145,"prepare.run_cpu_time_us":907,"prepare.run_wall_time_us":1693,"row_lock_wait_count":1,"row_lock_wait_us":850,"thread_start_us":59,"threads_started":1}]]}
I20260812 08:01:56.186594 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.39->[]
W20260812 08:01:56.186635 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 201) took 1333 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.186725 32090 rpcz_store.cc:269] 0812 08:01:54.853204 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.853274 (+    70us) service_pool.cc:224] Handling call
0812 08:01:56.185719 (+1332445us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:56.186413 (+   694us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:56.186418 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.186428 (+    10us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:56.186432 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.186447 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.186567 (+   120us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.186623 (+    56us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3063847168}
I20260812 08:01:56.186923 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.39->[]
W20260812 08:01:56.187009 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 203) took 785 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.187083 32089 rpcz_store.cc:269] 0812 08:01:55.401592 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.401686 (+    94us) service_pool.cc:224] Handling call
0812 08:01:56.186559 (+784873us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:56.186767 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:56.186774 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.186785 (+    11us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:56.186793 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.186810 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.186942 (+   132us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.186973 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804543232}
I20260812 08:01:56.187165 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.39->[]
W20260812 08:01:56.187214 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 204) took 234 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.187260 32084 rpcz_store.cc:269] 0812 08:01:55.952347 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.952457 (+   110us) service_pool.cc:224] Handling call
0812 08:01:56.186889 (+234432us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:56.187008 (+   119us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:56.187010 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.187015 (+     5us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:56.187018 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.187053 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.187170 (+   117us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.187202 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538667264}
W20260812 08:01:56.187420 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 202) took 836 ms (client timeout 50 ms). Trace:
I20260812 08:01:56.187425 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.39]   Dedup: 1.39->[]
W20260812 08:01:56.187477 32085 rpcz_store.cc:269] 0812 08:01:55.350843 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.350948 (+   105us) service_pool.cc:224] Handling call
0812 08:01:56.187122 (+836174us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:56.187253 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:56.187258 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.187266 (+     8us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:56.187273 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.187285 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.187389 (+   104us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.187409 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1922527104}
W20260812 08:01:56.187741 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 205) took 184 ms (client timeout 50 ms). Trace:
W20260812 08:01:56.187819 32087 rpcz_store.cc:269] 0812 08:01:56.003192 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.003331 (+   139us) service_pool.cc:224] Handling call
0812 08:01:56.187396 (+184065us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:56.187563 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 38
0812 08:01:56.187566 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:56.187574 (+     8us) raft_consensus.cc:1645] Marking committed up to 38
0812 08:01:56.187578 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.187591 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:56.187703 (+   112us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:56.187728 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422805376}
I20260812 08:01:56.370929 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.40]   Dedup: 1.39->[1.40-1.40]
W20260812 08:01:56.372154 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 981ms of latency in SegmentAllocator::Sync()
W20260812 08:01:56.974363 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18b00
I20260812 08:01:57.099870 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.099877 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 233) took 1125 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.100128 31939 rpcz_store.cc:269] 0812 08:01:55.973975 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.974035 (+    60us) service_pool.cc:224] Handling call
0812 08:01:55.974196 (+   161us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:55.974299 (+   103us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:55.974302 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:55.974445 (+   143us) write_op.cc:270] Start()
0812 08:01:55.974474 (+    29us) write_op.cc:276] Timestamp: P: 1786521715972697 usec, L: 0
0812 08:01:55.974834 (+   360us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:55.974872 (+    38us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:55.974878 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:55.974891 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:57.099585 (+1124694us) raft_consensus.cc:1685] finished
0812 08:01:57.099588 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.099847 (+   259us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:55.974789 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:55.974908 (+   119us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:55.974912 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:55.974914 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:55.975239 (+   325us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:55.975251 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:55.975255 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:55.975534 (+   279us) tablet.cc:701] Row locks acquired
0812 08:01:55.975536 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":208,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":813,"thread_start_us":109,"threads_started":1}]]}
I20260812 08:01:57.100382 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.100450 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 237) took 265 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.100677 31936 rpcz_store.cc:269] 0812 08:01:56.834865 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.834947 (+    82us) service_pool.cc:224] Handling call
0812 08:01:57.099719 (+264772us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.100144 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.100149 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.100162 (+    13us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.100166 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.100179 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.100404 (+   225us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:57.100435 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":608502144}
I20260812 08:01:57.100952 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.101099 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 236) took 625 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.101186 31942 rpcz_store.cc:269] 0812 08:01:56.476121 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.476204 (+    83us) service_pool.cc:224] Handling call
0812 08:01:57.100347 (+624143us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.100723 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.100728 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.100740 (+    12us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.100748 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.100762 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.101043 (+   281us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:57.101081 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434961152}
I20260812 08:01:57.101347 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.101446 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 238) took 215 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.101519 31935 rpcz_store.cc:269] 0812 08:01:56.885969 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.886079 (+   110us) service_pool.cc:224] Handling call
0812 08:01:57.100922 (+214843us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.101076 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.101082 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.101093 (+    11us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.101099 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.101116 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.101396 (+   280us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:57.101430 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493541760}
I20260812 08:01:57.101891 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.102046 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 234) took 1076 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.102095 31937 rpcz_store.cc:269] 0812 08:01:56.025120 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.025344 (+   224us) service_pool.cc:224] Handling call
0812 08:01:57.101301 (+1075957us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.101482 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.101485 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.101493 (+     8us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.101498 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.101512 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.101997 (+   485us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:57.102032 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2473809664}
W20260812 08:01:57.102322 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 235) took 677 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.102424 31941 rpcz_store.cc:269] 0812 08:01:56.424581 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.424742 (+   161us) service_pool.cc:224] Handling call
0812 08:01:57.101699 (+676957us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.102046 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.102049 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.102060 (+    11us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.102102 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.102117 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.102281 (+   164us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:57.102310 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1556403456}
I20260812 08:01:57.330215 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40]   Dedup: 1.40->[]
I20260812 08:01:57.334832 32682 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=38, attempt_no=0}) took 23276 ms. Trace:
I20260812 08:01:57.334988 32682 rpcz_store.cc:276] 0812 08:01:34.058198 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:34.058276 (+    78us) service_pool.cc:224] Handling call
0812 08:01:57.334811 (+23276535us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:54.782392 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:54.782529 (+   137us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:54.782535 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:54.782538 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:54.782962 (+   424us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:54.782970 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:54.782973 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:55.972679 (+1189706us) lock_manager.cc:598] Waited 1189259us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:55.972703 (+    24us) tablet.cc:701] Row locks acquired
0812 08:01:55.972707 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:01:55.972761 (+    54us) write_op.cc:270] Start()
0812 08:01:55.972791 (+    30us) write_op.cc:276] Timestamp: P: 1786521715972697 usec, L: 0
0812 08:01:55.972794 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:55.972988 (+   194us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:57.332049 (+1359061us) op_driver.cc:464] REPLICATION: finished
0812 08:01:57.332387 (+   338us) write_op.cc:301] APPLY: starting
0812 08:01:57.332423 (+    36us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:57.332718 (+   295us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:57.332720 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:57.334183 (+  1463us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:57.334249 (+    66us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:57.334255 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:57.334388 (+   133us) log.cc:844] Serialized 319 byte log entry
0812 08:01:57.334683 (+   295us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:57.334712 (+    29us) write_op.cc:454] Released schema lock
0812 08:01:57.334720 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":265,"num_ops":50,"prepare.queue_time_us":20723703,"prepare.run_cpu_time_us":1729,"prepare.run_wall_time_us":1190705,"raft.queue_time_us":354,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":340,"replication_time_us":1359203,"row_lock_wait_count":1,"row_lock_wait_us":1189259,"thread_start_us":91,"threads_started":1}]]}
W20260812 08:01:57.335750 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 970ms of latency in SegmentAllocator::Sync()
W20260812 08:01:57.336905 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 942ms of latency in SegmentAllocator::Sync()
W20260812 08:01:57.353832 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 206) took 983 ms (client timeout 50 ms). Trace:
I20260812 08:01:57.353860 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.40]   Dedup: 1.40->[]
I20260812 08:01:57.354249 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.353945 32087 rpcz_store.cc:269] 0812 08:01:56.370511 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.370655 (+   144us) service_pool.cc:224] Handling call
0812 08:01:56.370869 (+   214us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:56.371180 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:56.371391 (+   211us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:56.371509 (+   118us) write_op.cc:270] Start()
0812 08:01:56.371534 (+    25us) write_op.cc:276] Timestamp: P: 1786521715972697 usec, L: 0
0812 08:01:56.371850 (+   316us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:56.371891 (+    41us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:56.371896 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:56.371909 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:57.353671 (+981762us) raft_consensus.cc:1685] finished
0812 08:01:57.353674 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.353799 (+   125us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:56.371761 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:56.371878 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:56.371882 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:56.371884 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:56.372122 (+   238us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:56.372131 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:56.372134 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:56.373532 (+  1398us) tablet.cc:701] Row locks acquired
0812 08:01:56.373534 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":1841,"row_lock_wait_count":1,"row_lock_wait_us":1124,"thread_start_us":80,"threads_started":1}]]}
W20260812 08:01:57.354332 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 207) took 932 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.354377 32085 rpcz_store.cc:269] 0812 08:01:56.421822 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.421909 (+    87us) service_pool.cc:224] Handling call
0812 08:01:57.353809 (+931900us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:57.354114 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.354118 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.354129 (+    11us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.354133 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.354145 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.354284 (+   139us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:57.354318 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2142630144}
W20260812 08:01:57.354506 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 208) took 449 ms (client timeout 50 ms). Trace:
I20260812 08:01:57.354548 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.354542 32084 rpcz_store.cc:269] 0812 08:01:56.905141 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.905475 (+   334us) service_pool.cc:224] Handling call
0812 08:01:57.354218 (+448743us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:57.354349 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 39
0812 08:01:57.354352 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.354358 (+     6us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.354362 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.354371 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.354476 (+   105us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:57.354495 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1031110016}
I20260812 08:01:57.355089 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.40]   Dedup: 1.40->[]
W20260812 08:01:57.355327 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 209) took 399 ms (client timeout 50 ms). Trace:
W20260812 08:01:57.355372 32089 rpcz_store.cc:269] 0812 08:01:56.956390 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:56.956515 (+   125us) service_pool.cc:224] Handling call
0812 08:01:57.355064 (+398549us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:01:57.355185 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:57.355188 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:57.355193 (+     5us) raft_consensus.cc:1645] Marking committed up to 39
0812 08:01:57.355197 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.355205 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:57.355295 (+    90us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:57.355319 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":915662080}
W20260812 08:01:57.357569 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 874ms of latency in SegmentAllocator::Sync()
W20260812 08:01:57.436825 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:57.458012 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:58.232380 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 211) took 875 ms (client timeout 50 ms). Trace:
I20260812 08:01:58.232443 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.232523 32089 rpcz_store.cc:269] 0812 08:01:57.356653 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.356719 (+    66us) service_pool.cc:224] Handling call
0812 08:01:57.356883 (+   164us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.356973 (+    90us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:57.356976 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:57.357088 (+   112us) write_op.cc:270] Start()
0812 08:01:57.357109 (+    21us) write_op.cc:276] Timestamp: P: 1786521717334816 usec, L: 0
0812 08:01:57.357356 (+   247us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:57.357416 (+    60us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:57.357422 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.357433 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:58.232251 (+874818us) raft_consensus.cc:1685] finished
0812 08:01:58.232254 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.232346 (+    92us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:57.357477 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:57.357598 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:57.357602 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:57.357604 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:57.357924 (+   320us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:57.357938 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:57.357941 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:57.358241 (+   300us) tablet.cc:701] Row locks acquired
0812 08:01:57.358243 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":290,"prepare.run_cpu_time_us":878,"prepare.run_wall_time_us":877,"thread_start_us":75,"threads_started":1}]]}
I20260812 08:01:58.232787 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.232859 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 212) took 824 ms (client timeout 50 ms). Trace:
W20260812 08:01:58.232930 32090 rpcz_store.cc:269] 0812 08:01:57.407934 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.408024 (+    90us) service_pool.cc:224] Handling call
0812 08:01:58.232392 (+824368us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.232616 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.232621 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.232635 (+    14us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.232641 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.232666 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.232787 (+   121us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:58.232823 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1895396608}
I20260812 08:01:58.233078 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.233108 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 214) took 342 ms (client timeout 50 ms). Trace:
W20260812 08:01:58.233326 32085 rpcz_store.cc:269] 0812 08:01:57.891092 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.891247 (+   155us) service_pool.cc:224] Handling call
0812 08:01:58.232751 (+341504us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.232903 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.232909 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.232919 (+    10us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.232923 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.232934 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.233068 (+   134us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:58.233095 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":784901504}
I20260812 08:01:58.233474 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.233521 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 213) took 394 ms (client timeout 50 ms). Trace:
W20260812 08:01:58.233603 32084 rpcz_store.cc:269] 0812 08:01:57.839578 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.839796 (+   218us) service_pool.cc:224] Handling call
0812 08:01:58.233049 (+393253us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.233345 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.233349 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.233356 (+     7us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.233360 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.233370 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.233480 (+   110us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:58.233508 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903994112}
I20260812 08:01:58.279549 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.279549 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 241) took 943 ms (client timeout 50 ms). Trace:
I20260812 08:01:58.280148 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.279932 31941 rpcz_store.cc:269] 0812 08:01:57.335979 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.336034 (+    55us) service_pool.cc:224] Handling call
0812 08:01:57.336161 (+   127us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:57.336232 (+    71us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:57.336234 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:57.336384 (+   150us) write_op.cc:270] Start()
0812 08:01:57.336406 (+    22us) write_op.cc:276] Timestamp: P: 1786521717334816 usec, L: 0
0812 08:01:57.336643 (+   237us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:57.336676 (+    33us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:57.336680 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:57.336690 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:58.279377 (+942687us) raft_consensus.cc:1685] finished
0812 08:01:58.279380 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.279521 (+   141us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:57.336772 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:57.336879 (+   107us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:57.336881 (+     2us) write_op.cc:435] Acquired schema lock
0812 08:01:57.336882 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:57.337372 (+   490us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:57.337381 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:57.337384 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:57.337564 (+   180us) tablet.cc:701] Row locks acquired
0812 08:01:57.337565 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":309,"prepare.run_cpu_time_us":849,"prepare.run_wall_time_us":847,"thread_start_us":68,"threads_started":1}]]}
W20260812 08:01:58.280314 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 243) took 460 ms (client timeout 50 ms). Trace:
W20260812 08:01:58.280846 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 242) took 894 ms (client timeout 50 ms). Trace:
I20260812 08:01:58.280988 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41]   Dedup: 1.41->[]
W20260812 08:01:58.280968 31937 rpcz_store.cc:269] 0812 08:01:57.386865 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.386980 (+   115us) service_pool.cc:224] Handling call
0812 08:01:58.280113 (+893133us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.280478 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.280481 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.280490 (+     9us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.280494 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.280506 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.280792 (+   286us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:58.280828 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053548416}
W20260812 08:01:58.280867 31935 rpcz_store.cc:269] 0812 08:01:57.819755 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.819937 (+   182us) service_pool.cc:224] Handling call
0812 08:01:58.279492 (+459555us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.279936 (+   444us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.279940 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.279957 (+    17us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.279964 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.279979 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.280202 (+   223us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:58.280235 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056377728}
W20260812 08:01:58.281504 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 244) took 410 ms (client timeout 50 ms). Trace:
W20260812 08:01:58.281580 31942 rpcz_store.cc:269] 0812 08:01:57.870853 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.870972 (+   119us) service_pool.cc:224] Handling call
0812 08:01:58.280934 (+409962us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.281317 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 40
0812 08:01:58.281321 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:58.281331 (+    10us) raft_consensus.cc:1645] Marking committed up to 40
0812 08:01:58.281336 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.281350 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:58.281461 (+   111us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:58.281491 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":942357248}
I20260812 08:01:58.309134 32689 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=39, attempt_no=0}) took 23172 ms. Trace:
I20260812 08:01:58.309507 32689 rpcz_store.cc:276] 0812 08:01:35.136903 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:35.136993 (+    90us) service_pool.cc:224] Handling call
0812 08:01:58.309110 (+23172117us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:55.973145 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:55.973289 (+   144us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:55.973294 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:55.973298 (+     4us) tablet.cc:662] Decoding operations
0812 08:01:55.973748 (+   450us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:55.973759 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:55.973764 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:57.334842 (+1361078us) lock_manager.cc:598] Waited 1360619us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:57.334856 (+    14us) tablet.cc:701] Row locks acquired
0812 08:01:57.334860 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:01:57.334903 (+    43us) write_op.cc:270] Start()
0812 08:01:57.334930 (+    27us) write_op.cc:276] Timestamp: P: 1786521717334816 usec, L: 0
0812 08:01:57.334933 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:57.335072 (+   139us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:58.306346 (+971274us) op_driver.cc:464] REPLICATION: finished
0812 08:01:58.306763 (+   417us) write_op.cc:301] APPLY: starting
0812 08:01:58.306800 (+    37us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:58.306854 (+    54us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:58.306855 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:58.308314 (+  1459us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:58.308369 (+    55us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:58.308374 (+     5us) write_op.cc:312] APPLY: finished
0812 08:01:58.308504 (+   130us) log.cc:844] Serialized 319 byte log entry
0812 08:01:58.308954 (+   450us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:58.308993 (+    39us) write_op.cc:454] Released schema lock
0812 08:01:58.309003 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"num_ops":50,"prepare.queue_time_us":20835780,"prepare.run_cpu_time_us":1620,"prepare.run_wall_time_us":1362006,"raft.queue_time_us":161,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":1040,"replication_time_us":971361,"row_lock_wait_count":1,"row_lock_wait_us":1360619,"spinlock_wait_cycles":12544,"thread_start_us":107,"threads_started":1}]]}
W20260812 08:01:58.310261 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 995ms of latency in SegmentAllocator::Sync()
W20260812 08:01:58.312475 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 948ms of latency in SegmentAllocator::Sync()
I20260812 08:01:58.339268 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.42]   Dedup: 1.41->[1.42-1.42]
W20260812 08:01:58.341504 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1077ms of latency in SegmentAllocator::Sync()
W20260812 08:01:58.411882 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:59.261163 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 217) took 950 ms (client timeout 50 ms). Trace:
I20260812 08:01:59.261200 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.261291 32087 rpcz_store.cc:269] 0812 08:01:58.311214 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.311338 (+   124us) service_pool.cc:224] Handling call
0812 08:01:58.311527 (+   189us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:58.311602 (+    75us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:58.311604 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:58.311720 (+   116us) write_op.cc:270] Start()
0812 08:01:58.311749 (+    29us) write_op.cc:276] Timestamp: P: 1786521718308951 usec, L: 0
0812 08:01:58.312188 (+   439us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:58.312237 (+    49us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:58.312245 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.312258 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:59.261015 (+948757us) raft_consensus.cc:1685] finished
0812 08:01:59.261018 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.261126 (+   108us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:58.312050 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:58.312161 (+   111us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:58.312166 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:58.312168 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:58.312500 (+   332us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:58.312512 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:58.312514 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:58.312904 (+   390us) tablet.cc:701] Row locks acquired
0812 08:01:58.312907 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":924,"prepare.run_wall_time_us":925,"thread_start_us":126,"threads_started":1}]]}
I20260812 08:01:59.261685 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.261770 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 218) took 899 ms (client timeout 50 ms). Trace:
I20260812 08:01:59.261927 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.261965 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 219) took 491 ms (client timeout 50 ms). Trace:
W20260812 08:01:59.261852 32084 rpcz_store.cc:269] 0812 08:01:58.362024 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.362240 (+   216us) service_pool.cc:224] Handling call
0812 08:01:59.261145 (+898905us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:59.261573 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.261576 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.261586 (+    10us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.261589 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.261599 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.261723 (+   124us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:59.261756 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066761472}
W20260812 08:01:59.262168 32085 rpcz_store.cc:269] 0812 08:01:58.770660 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.770740 (+    80us) service_pool.cc:224] Handling call
0812 08:01:59.261657 (+490917us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:59.261786 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.261789 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.261796 (+     7us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.261800 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.261808 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.261928 (+   120us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:59.261953 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1128541440}
W20260812 08:01:59.262315 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 220) took 440 ms (client timeout 50 ms). Trace:
W20260812 08:01:59.262368 32090 rpcz_store.cc:269] 0812 08:01:58.822160 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.822259 (+    99us) service_pool.cc:224] Handling call
0812 08:01:59.261897 (+439638us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:59.262134 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.262138 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.262146 (+     8us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.262150 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.262162 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.262279 (+   117us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:59.262305 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1010616064}
I20260812 08:01:59.262375 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.262847 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 221) took 40 ms (client timeout 50 ms). Trace:
W20260812 08:01:59.262918 32089 rpcz_store.cc:269] 0812 08:01:59.222185 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.222327 (+   142us) service_pool.cc:224] Handling call
0812 08:01:59.262347 (+ 40020us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:59.262665 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.262668 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.262675 (+     7us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.262679 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.262688 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.262812 (+   124us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:01:59.262836 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91625216}
I20260812 08:01:59.308931 32695 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=40, attempt_no=0}) took 23034 ms. Trace:
I20260812 08:01:59.309113 32695 rpcz_store.cc:276] 0812 08:01:36.274021 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:36.274162 (+   141us) service_pool.cc:224] Handling call
0812 08:01:59.308908 (+23034746us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:57.335169 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:57.335313 (+   144us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:57.335318 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:01:57.335320 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:57.335620 (+   300us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:57.335628 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:57.335631 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:58.309004 (+973373us) lock_manager.cc:598] Waited 973046us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:58.309016 (+    12us) tablet.cc:701] Row locks acquired
0812 08:01:58.309018 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:58.309068 (+    50us) write_op.cc:270] Start()
0812 08:01:58.309093 (+    25us) write_op.cc:276] Timestamp: P: 1786521718308951 usec, L: 0
0812 08:01:58.309095 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:01:58.309498 (+   403us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:59.306197 (+996699us) op_driver.cc:464] REPLICATION: finished
0812 08:01:59.306847 (+   650us) write_op.cc:301] APPLY: starting
0812 08:01:59.306896 (+    49us) tablet.cc:1367] starting BulkCheckPresence
0812 08:01:59.306956 (+    60us) tablet.cc:1370] finished BulkCheckPresence
0812 08:01:59.306957 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:01:59.308470 (+  1513us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:01:59.308527 (+    57us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:01:59.308533 (+     6us) write_op.cc:312] APPLY: finished
0812 08:01:59.308683 (+   150us) log.cc:844] Serialized 319 byte log entry
0812 08:01:59.308764 (+    81us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:01:59.308795 (+    31us) write_op.cc:454] Released schema lock
0812 08:01:59.308806 (+    11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":512,"num_ops":50,"prepare.queue_time_us":21060520,"prepare.run_cpu_time_us":1421,"prepare.run_wall_time_us":974442,"raft.queue_time_us":646,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":211,"replication_time_us":997006,"row_lock_wait_count":1,"row_lock_wait_us":973046,"thread_start_us":431,"threads_started":1}]]}
W20260812 08:01:59.309844 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1033ms of latency in SegmentAllocator::Sync()
W20260812 08:01:59.311127 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 927ms of latency in SegmentAllocator::Sync()
W20260812 08:01:59.360900 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 20 similar messages]
W20260812 08:01:59.412590 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:01:59.419394 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 245) took 1080 ms (client timeout 50 ms). Trace:
I20260812 08:01:59.419394 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.42]   Dedup: 1.42->[]
I20260812 08:01:59.419771 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.420104 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 248) took 412 ms (client timeout 50 ms). Trace:
I20260812 08:01:59.420346 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.42]   Dedup: 1.42->[]
W20260812 08:01:59.420308 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 247) took 463 ms (client timeout 50 ms). Trace:
I20260812 08:01:59.420692 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.43]   Dedup: 1.42->[1.43-1.43]
W20260812 08:01:59.420737 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 246) took 1030 ms (client timeout 50 ms). Trace:
W20260812 08:01:59.420624 31937 rpcz_store.cc:269] 0812 08:01:58.956781 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.956908 (+   127us) service_pool.cc:224] Handling call
0812 08:01:59.419740 (+462832us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:59.420103 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.420106 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.420117 (+    11us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.420125 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.420137 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.420270 (+   133us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:59.420296 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1063967104}
W20260812 08:01:59.420785 31935 rpcz_store.cc:269] 0812 08:01:58.389862 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.389974 (+   112us) service_pool.cc:224] Handling call
0812 08:01:59.420304 (+1030330us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:59.420512 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.420516 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.420525 (+     9us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.420529 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.420545 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.420693 (+   148us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:59.420724 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2369043840}
W20260812 08:01:59.419597 31942 rpcz_store.cc:269] 0812 08:01:58.338779 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.338960 (+   181us) service_pool.cc:224] Handling call
0812 08:01:58.339200 (+   240us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:58.340149 (+   949us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:58.340575 (+   426us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:58.340712 (+   137us) write_op.cc:270] Start()
0812 08:01:58.340747 (+    35us) write_op.cc:276] Timestamp: P: 1786521718308951 usec, L: 0
0812 08:01:58.341103 (+   356us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:58.341146 (+    43us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:58.341151 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:58.341166 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:01:59.419231 (+1078065us) raft_consensus.cc:1685] finished
0812 08:01:59.419234 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.419356 (+   122us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:58.340993 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:58.341117 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:58.341123 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:58.341125 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:58.341368 (+   243us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:58.341376 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:58.341378 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:58.342857 (+  1479us) tablet.cc:701] Row locks acquired
0812 08:01:58.342859 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:59.420866 (+1078007us) op_driver.cc:464] REPLICATION: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":869,"prepare.run_wall_time_us":2046,"replication_time_us":1080123,"row_lock_wait_count":1,"row_lock_wait_us":1287,"spinlock_wait_cycles":220928,"thread_start_us":197,"threads_started":2}]]}
W20260812 08:01:59.420158 31941 rpcz_store.cc:269] 0812 08:01:59.008064 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.008243 (+   179us) service_pool.cc:224] Handling call
0812 08:01:59.419330 (+411087us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:01:59.419657 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 41
0812 08:01:59.419661 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:01:59.419673 (+    12us) raft_consensus.cc:1645] Marking committed up to 41
0812 08:01:59.419677 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.419692 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:01:59.420043 (+   351us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:01:59.420087 (+    44us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944882048}
W20260812 08:01:59.421823 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1044ms of latency in SegmentAllocator::Sync()
W20260812 08:01:59.870782 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
I20260812 08:02:00.238891 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.238897 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 223) took 928 ms (client timeout 50 ms). Trace:
W20260812 08:02:00.239179 32089 rpcz_store.cc:269] 0812 08:01:59.310153 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.310222 (+    69us) service_pool.cc:224] Handling call
0812 08:01:59.310372 (+   150us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:01:59.310449 (+    77us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:01:59.310453 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:59.310615 (+   162us) write_op.cc:270] Start()
0812 08:01:59.310642 (+    27us) write_op.cc:276] Timestamp: P: 1786521719308862 usec, L: 0
0812 08:01:59.310943 (+   301us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:59.310963 (+    20us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:01:59.310968 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.310981 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:00.238624 (+927643us) raft_consensus.cc:1685] finished
0812 08:02:00.238627 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.238859 (+   232us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:59.311070 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:59.311167 (+    97us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:59.311169 (+     2us) write_op.cc:435] Acquired schema lock
0812 08:01:59.311170 (+     1us) tablet.cc:662] Decoding operations
0812 08:01:59.311481 (+   311us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:59.311489 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:59.311491 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:59.311668 (+   177us) tablet.cc:701] Row locks acquired
0812 08:01:59.311669 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":355,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":653,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:02:00.239627 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.239729 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 224) took 877 ms (client timeout 50 ms). Trace:
W20260812 08:02:00.239799 32090 rpcz_store.cc:269] 0812 08:01:59.362042 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.362192 (+   150us) service_pool.cc:224] Handling call
0812 08:02:00.238829 (+876637us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:00.239149 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:00.239152 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:00.239163 (+    11us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:00.239166 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.239176 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.239665 (+   489us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:00.239710 (+    45us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2015576320}
I20260812 08:02:00.239931 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.239972 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 225) took 456 ms (client timeout 50 ms). Trace:
W20260812 08:02:00.240033 32085 rpcz_store.cc:269] 0812 08:01:59.783793 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.783875 (+    82us) service_pool.cc:224] Handling call
0812 08:02:00.239594 (+455719us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:00.239762 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:00.239765 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:00.239774 (+     9us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:00.239778 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.239789 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.239930 (+   141us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:00.239958 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1047525120}
W20260812 08:02:00.240216 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 226) took 405 ms (client timeout 50 ms). Trace:
W20260812 08:02:00.241097 32084 rpcz_store.cc:269] 0812 08:01:59.835014 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.835120 (+   106us) service_pool.cc:224] Handling call
0812 08:02:00.239901 (+404781us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:00.240021 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:00.240027 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:00.240039 (+    12us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:00.240046 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.240061 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.240178 (+   117us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:00.240203 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":930455424}
I20260812 08:02:00.242342 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.310282 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0000
W20260812 08:02:00.347142 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 996ms of latency in SegmentAllocator::Sync()
I20260812 08:02:00.347751 32702 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=41, attempt_no=0}) took 23030 ms. Trace:
I20260812 08:02:00.347852 32702 rpcz_store.cc:276] 0812 08:01:37.317791 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:37.317889 (+    98us) service_pool.cc:224] Handling call
0812 08:02:00.347727 (+23029838us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:58.309721 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:58.309851 (+   130us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:58.309859 (+     8us) write_op.cc:435] Acquired schema lock
0812 08:01:58.309861 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:58.310339 (+   478us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:58.310352 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:58.310358 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:59.308868 (+998510us) lock_manager.cc:598] Waited 998085us for lock on <redacted> (key hash 11785448468237383701)
0812 08:01:59.308884 (+    16us) tablet.cc:701] Row locks acquired
0812 08:01:59.308886 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:01:59.308932 (+    46us) write_op.cc:270] Start()
0812 08:01:59.308959 (+    27us) write_op.cc:276] Timestamp: P: 1786521719308862 usec, L: 0
0812 08:01:59.308962 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:01:59.309106 (+   144us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:00.343647 (+1034541us) op_driver.cc:464] REPLICATION: finished
0812 08:02:00.344084 (+   437us) write_op.cc:301] APPLY: starting
0812 08:02:00.344133 (+    49us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:00.344202 (+    69us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:00.344203 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:00.346146 (+  1943us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:00.346219 (+    73us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:00.346229 (+    10us) write_op.cc:312] APPLY: finished
0812 08:02:00.346411 (+   182us) log.cc:844] Serialized 319 byte log entry
0812 08:02:00.346500 (+    89us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:00.347630 (+  1130us) write_op.cc:454] Released schema lock
0812 08:02:00.347642 (+    12us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":349,"num_ops":50,"prepare.queue_time_us":20991367,"prepare.run_cpu_time_us":1395,"prepare.run_wall_time_us":999479,"raft.queue_time_us":296,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":191,"replication_time_us":1034617,"row_lock_wait_count":1,"row_lock_wait_us":998085,"spinlock_wait_cycles":768,"thread_start_us":164,"threads_started":1}]]}
W20260812 08:02:00.348856 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1097ms of latency in SegmentAllocator::Sync()
W20260812 08:02:00.450256 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:00.466784 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 249) took 1086 ms (client timeout 50 ms). Trace:
I20260812 08:02:00.466741 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.467078 31936 rpcz_store.cc:269] 0812 08:01:59.380488 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.380619 (+   131us) service_pool.cc:224] Handling call
0812 08:01:59.420611 (+ 39992us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:01:59.420815 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:01:59.421030 (+   215us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:01:59.421192 (+   162us) write_op.cc:270] Start()
0812 08:01:59.421227 (+    35us) write_op.cc:276] Timestamp: P: 1786521719308862 usec, L: 0
0812 08:01:59.421568 (+   341us) log.cc:844] Serialized 2464 byte log entry
0812 08:01:59.421613 (+    45us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:01:59.421619 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:01:59.421633 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:00.466387 (+1044754us) raft_consensus.cc:1685] finished
0812 08:02:00.466390 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.466700 (+   310us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:00.466761 (+    61us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:59.421742 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:59.421882 (+   140us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:59.421888 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:01:59.421891 (+     3us) tablet.cc:662] Decoding operations
0812 08:01:59.422727 (+   836us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:59.422742 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:59.422745 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:01:59.423266 (+   521us) tablet.cc:701] Row locks acquired
0812 08:01:59.423268 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":90973952,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":410,"prepare.run_cpu_time_us":1510,"prepare.run_wall_time_us":1595,"row_lock_wait_count":1,"row_lock_wait_us":304,"thread_start_us":64,"threads_started":1}]]}
W20260812 08:02:00.467374 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 252) took 646 ms (client timeout 50 ms). Trace:
I20260812 08:02:00.467376 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.43]   Dedup: 1.43->[]
W20260812 08:02:00.467461 31937 rpcz_store.cc:269] 0812 08:01:59.820707 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.820992 (+   285us) service_pool.cc:224] Handling call
0812 08:02:00.466529 (+645537us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:00.467161 (+   632us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:00.467165 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:00.467178 (+    13us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:00.467182 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.467203 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.467335 (+   132us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:00.467361 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1483478272}
W20260812 08:02:00.467881 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 251) took 699 ms (client timeout 50 ms). Trace:
I20260812 08:02:00.467952 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.44]   Dedup: 1.43->[1.44-1.44]
W20260812 08:02:00.467980 31942 rpcz_store.cc:269] 0812 08:01:59.768841 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.768964 (+   123us) service_pool.cc:224] Handling call
0812 08:02:00.467335 (+698371us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:00.467588 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:00.467593 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:00.467605 (+    12us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:00.467612 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.467640 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:00.467765 (+   125us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:00.467797 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1605382016}
W20260812 08:02:00.469070 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 999ms of latency in SegmentAllocator::Sync()
W20260812 08:02:01.347945 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1600
I20260812 08:02:01.446542 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.447090 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 229) took 1099 ms (client timeout 50 ms). Trace:
I20260812 08:02:01.447273 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.447388 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 232) took 592 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.447576 32089 rpcz_store.cc:269] 0812 08:02:00.855034 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.855139 (+   105us) service_pool.cc:224] Handling call
0812 08:02:01.446486 (+591347us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:01.447027 (+   541us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.447030 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.447041 (+    11us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.447044 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.447055 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.447334 (+   279us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:01.447376 (+    42us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359382784}
I20260812 08:02:01.448028 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.447197 32084 rpcz_store.cc:269] 0812 08:02:00.347766 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.347825 (+    59us) service_pool.cc:224] Handling call
0812 08:02:00.347993 (+   168us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:00.348079 (+    86us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:00.348082 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:00.348254 (+   172us) write_op.cc:270] Start()
0812 08:02:00.348290 (+    36us) write_op.cc:276] Timestamp: P: 1786521720346566 usec, L: 0
0812 08:02:00.348632 (+   342us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:00.348664 (+    32us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:00.348670 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.348683 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:01.446395 (+1097712us) raft_consensus.cc:1685] finished
0812 08:02:01.446398 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.447053 (+   655us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:00.348563 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:00.348679 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:00.348685 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:00.348689 (+     4us) tablet.cc:662] Decoding operations
0812 08:02:00.349028 (+   339us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:00.349041 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:00.349045 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:00.349340 (+   295us) tablet.cc:701] Row locks acquired
0812 08:02:00.349343 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":851,"prepare.run_wall_time_us":852,"thread_start_us":89,"threads_started":1}]]}
W20260812 08:02:01.448454 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 231) took 644 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.448601 32090 rpcz_store.cc:269] 0812 08:02:00.803834 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.803967 (+   133us) service_pool.cc:224] Handling call
0812 08:02:01.447993 (+644026us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:01.448163 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.448167 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.448248 (+    81us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.448257 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.448269 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.448406 (+   137us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:01.448438 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1480118656}
I20260812 08:02:01.448498 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.449162 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 230) took 1049 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.449245 32085 rpcz_store.cc:269] 0812 08:02:00.400144 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.400230 (+    86us) service_pool.cc:224] Handling call
0812 08:02:01.448466 (+1048236us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:01.448976 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.448979 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.448986 (+     7us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.448993 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.449008 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.449122 (+   114us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:01.449149 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2410272000}
I20260812 08:02:01.451648 32711 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=42, attempt_no=0}) took 22778 ms. Trace:
I20260812 08:02:01.451889 32711 rpcz_store.cc:276] 0812 08:01:38.673663 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:38.673795 (+   132us) service_pool.cc:224] Handling call
0812 08:02:01.451627 (+22777832us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:01:59.309230 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:01:59.309351 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:01:59.309355 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:01:59.309357 (+     2us) tablet.cc:662] Decoding operations
0812 08:01:59.309866 (+   509us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:01:59.309875 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:01:59.309879 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:00.346553 (+1036674us) lock_manager.cc:598] Waited 1036411us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:00.346576 (+    23us) tablet.cc:701] Row locks acquired
0812 08:02:00.346604 (+    28us) write_op.cc:260] PREPARE: finished
0812 08:02:00.346664 (+    60us) write_op.cc:270] Start()
0812 08:02:00.346690 (+    26us) write_op.cc:276] Timestamp: P: 1786521720346566 usec, L: 0
0812 08:02:00.346692 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:00.346870 (+   178us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:01.448711 (+1101841us) op_driver.cc:464] REPLICATION: finished
0812 08:02:01.448943 (+   232us) write_op.cc:301] APPLY: starting
0812 08:02:01.448987 (+    44us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:01.449616 (+   629us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:01.449618 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:01.451215 (+  1597us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:01.451277 (+    62us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:01.451283 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:01.451415 (+   132us) log.cc:844] Serialized 319 byte log entry
0812 08:02:01.451496 (+    81us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:01.451532 (+    36us) write_op.cc:454] Released schema lock
0812 08:02:01.451541 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"num_ops":50,"prepare.queue_time_us":20634993,"prepare.run_cpu_time_us":1572,"prepare.run_wall_time_us":1037753,"raft.queue_time_us":726,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"replication_time_us":1101965,"row_lock_wait_count":1,"row_lock_wait_us":1036411,"thread_start_us":93,"threads_started":1}]]}
W20260812 08:02:01.452464 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 959ms of latency in SegmentAllocator::Sync()
W20260812 08:02:01.454049 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 941ms of latency in SegmentAllocator::Sync()
W20260812 08:02:01.469142 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 254) took 1048 ms (client timeout 50 ms). Trace:
I20260812 08:02:01.469219 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.43]   Dedup: 1.43->[]
W20260812 08:02:01.469663 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 250) took 2038 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.469297 31939 rpcz_store.cc:269] 0812 08:02:00.420556 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.420761 (+   205us) service_pool.cc:224] Handling call
0812 08:02:00.467775 (+ 47014us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:00.468055 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:00.468301 (+   246us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:00.468404 (+   103us) write_op.cc:270] Start()
0812 08:02:00.468431 (+    27us) write_op.cc:276] Timestamp: P: 1786521720346566 usec, L: 0
0812 08:02:00.468773 (+   342us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:00.468822 (+    49us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:00.468828 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:00.468841 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:01.468626 (+999785us) raft_consensus.cc:1685] finished
0812 08:02:01.468628 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.469058 (+   430us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:01.469115 (+    57us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:00.468754 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:00.468992 (+   238us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:00.469001 (+     9us) write_op.cc:435] Acquired schema lock
0812 08:02:00.469004 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:00.469450 (+   446us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:00.469460 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:00.469463 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:00.470118 (+   655us) tablet.cc:701] Row locks acquired
0812 08:02:00.470122 (+     4us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":107412352,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":217,"prepare.run_cpu_time_us":1152,"prepare.run_wall_time_us":1459,"row_lock_wait_count":1,"row_lock_wait_us":437,"thread_start_us":97,"threads_started":1}]]}
I20260812 08:02:01.469733 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.469755 31941 rpcz_store.cc:269] 0812 08:01:59.431043 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.431211 (+   168us) service_pool.cc:224] Handling call
0812 08:02:01.469174 (+2037963us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:01.469349 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 42
0812 08:02:01.469395 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.469410 (+    15us) raft_consensus.cc:1645] Marking committed up to 42
0812 08:02:01.469416 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.469452 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.469577 (+   125us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:01.469607 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4685890304}
W20260812 08:02:01.470026 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 253) took 1100 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.470158 31935 rpcz_store.cc:269] 0812 08:02:00.369393 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.369502 (+   109us) service_pool.cc:224] Handling call
0812 08:02:01.469694 (+1100192us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:01.469851 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.469855 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.469862 (+     7us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.469866 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.469875 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.469986 (+   111us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:01.470012 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2529296768}
I20260812 08:02:01.470036 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.44]   Dedup: 1.44->[]
I20260812 08:02:01.470909 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.44]   Dedup: 1.44->[]
W20260812 08:02:01.471011 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 256) took 486 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.471231 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 255) took 538 ms (client timeout 50 ms). Trace:
W20260812 08:02:01.471194 31937 rpcz_store.cc:269] 0812 08:02:00.984219 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.984391 (+   172us) service_pool.cc:224] Handling call
0812 08:02:01.470006 (+485615us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:01.470680 (+   674us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.470687 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.470699 (+    12us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.470705 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.470716 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.470828 (+   112us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:01.470853 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116220672}
W20260812 08:02:01.471418 31942 rpcz_store.cc:269] 0812 08:02:00.933111 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.933176 (+    65us) service_pool.cc:224] Handling call
0812 08:02:01.470876 (+537700us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:01.471022 (+   146us) raft_consensus.cc:1534] Early marking committed up to index 43
0812 08:02:01.471027 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:01.471038 (+    11us) raft_consensus.cc:1645] Marking committed up to 43
0812 08:02:01.471044 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.471056 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:01.471183 (+   127us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:01.471215 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235751168}
I20260812 08:02:01.491627 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.45]   Dedup: 1.44->[1.45-1.45]
W20260812 08:02:01.493290 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1058ms of latency in SegmentAllocator::Sync()
W20260812 08:02:01.554791 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:02.395828 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 235) took 942 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.395982 32090 rpcz_store.cc:269] 0812 08:02:01.452948 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.453001 (+    53us) service_pool.cc:224] Handling call
0812 08:02:01.453158 (+   157us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:01.453247 (+    89us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:01.453250 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:01.453390 (+   140us) write_op.cc:270] Start()
0812 08:02:01.453419 (+    29us) write_op.cc:276] Timestamp: P: 1786521721451820 usec, L: 0
0812 08:02:01.453837 (+   418us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:01.454194 (+   357us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:01.454202 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.454218 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:02.395689 (+941471us) raft_consensus.cc:1685] finished
0812 08:02:02.395692 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.395795 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:01.453876 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:01.453992 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:01.453995 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:01.453996 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:01.454390 (+   394us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:01.454412 (+    22us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:01.454415 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:01.454889 (+   474us) tablet.cc:701] Row locks acquired
0812 08:02:01.454892 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":368,"prepare.run_cpu_time_us":1097,"prepare.run_wall_time_us":1096,"thread_start_us":104,"threads_started":1}]]}
I20260812 08:02:02.395934 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45]   Dedup: 1.45->[]
I20260812 08:02:02.396512 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45]   Dedup: 1.45->[]
W20260812 08:02:02.396606 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 236) took 892 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.396657 32085 rpcz_store.cc:269] 0812 08:02:01.504622 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.504752 (+   130us) service_pool.cc:224] Handling call
0812 08:02:02.395885 (+891133us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:02.396360 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.396367 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.396383 (+    16us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.396390 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.396404 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.396549 (+   145us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:02.396589 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2048947456}
I20260812 08:02:02.396785 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45]   Dedup: 1.45->[]
W20260812 08:02:02.396836 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 237) took 474 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.396893 32084 rpcz_store.cc:269] 0812 08:02:01.922165 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.922286 (+   121us) service_pool.cc:224] Handling call
0812 08:02:02.396478 (+474192us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:02.396643 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.396646 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.396655 (+     9us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.396659 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.396672 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.396802 (+   130us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:02.396825 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090130304}
I20260812 08:02:02.397069 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45]   Dedup: 1.45->[]
W20260812 08:02:02.397213 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 238) took 424 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.397271 32089 rpcz_store.cc:269] 0812 08:02:01.973259 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.973381 (+   122us) service_pool.cc:224] Handling call
0812 08:02:02.396752 (+423371us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:02.396926 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.396933 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.396944 (+    11us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.396948 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.396959 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.397136 (+   177us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:02.397196 (+    60us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":973207808}
W20260812 08:02:02.414978 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1162ms of latency in SegmentAllocator::Sync()
I20260812 08:02:02.415513 32720 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=43, attempt_no=0}) took 22739 ms. Trace:
I20260812 08:02:02.415663 32720 rpcz_store.cc:276] 0812 08:01:39.676285 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:39.676398 (+   113us) service_pool.cc:224] Handling call
0812 08:02:02.415498 (+22739100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:00.347021 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:00.347149 (+   128us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:00.347166 (+    17us) write_op.cc:435] Acquired schema lock
0812 08:02:00.347168 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:00.347473 (+   305us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:00.347485 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:00.347490 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:01.451736 (+1104246us) lock_manager.cc:598] Waited 1103944us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:01.451752 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:01.451754 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:01.451806 (+    52us) write_op.cc:270] Start()
0812 08:02:01.451831 (+    25us) write_op.cc:276] Timestamp: P: 1786521721451820 usec, L: 0
0812 08:02:01.451832 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:02:01.452058 (+   226us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:02.412112 (+960054us) op_driver.cc:464] REPLICATION: finished
0812 08:02:02.412515 (+   403us) write_op.cc:301] APPLY: starting
0812 08:02:02.412565 (+    50us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:02.412646 (+    81us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:02.412648 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:02.414118 (+  1470us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:02.414174 (+    56us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:02.414180 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:02.414331 (+   151us) log.cc:844] Serialized 319 byte log entry
0812 08:02:02.414403 (+    72us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:02.415409 (+  1006us) write_op.cc:454] Released schema lock
0812 08:02:02.415418 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":315,"num_ops":50,"prepare.queue_time_us":20670181,"prepare.run_cpu_time_us":1645,"prepare.run_wall_time_us":1105352,"raft.queue_time_us":369,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":202,"replication_time_us":960224,"row_lock_wait_count":1,"row_lock_wait_us":1103944,"thread_start_us":119,"threads_started":1}]]}
W20260812 08:02:02.416363 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 948ms of latency in SegmentAllocator::Sync()
W20260812 08:02:02.516884 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:02.554489 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 257) took 1063 ms (client timeout 50 ms). Trace:
I20260812 08:02:02.554489 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.45]   Dedup: 1.45->[]
W20260812 08:02:02.554706 31942 rpcz_store.cc:269] 0812 08:02:01.491283 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.491358 (+    75us) service_pool.cc:224] Handling call
0812 08:02:01.491567 (+   209us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:01.491814 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:01.492127 (+   313us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:01.492276 (+   149us) write_op.cc:270] Start()
0812 08:02:01.492304 (+    28us) write_op.cc:276] Timestamp: P: 1786521721451820 usec, L: 0
0812 08:02:01.492925 (+   621us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:01.492984 (+    59us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:01.492991 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:01.493007 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:02.554285 (+1061278us) raft_consensus.cc:1685] finished
0812 08:02:02.554295 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.554455 (+   160us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:01.492634 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:01.492750 (+   116us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:01.492754 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:01.492755 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:01.493051 (+   296us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:01.493065 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:01.493068 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:01.494164 (+  1096us) tablet.cc:701] Row locks acquired
0812 08:02:01.494166 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":234,"prepare.run_cpu_time_us":903,"prepare.run_wall_time_us":1599,"row_lock_wait_count":1,"row_lock_wait_us":762,"thread_start_us":248,"threads_started":1}]]}
I20260812 08:02:02.554934 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.45]   Dedup: 1.45->[]
I20260812 08:02:02.555217 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.45]   Dedup: 1.45->[]
W20260812 08:02:02.555285 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 259) took 632 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.555478 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 260) took 580 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.555558 31941 rpcz_store.cc:269] 0812 08:02:01.974941 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.975055 (+   114us) service_pool.cc:224] Handling call
0812 08:02:02.554885 (+579830us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:02.555080 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.555086 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.555098 (+    12us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.555107 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.555122 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.555447 (+   325us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:02.555469 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1332856192}
I20260812 08:02:02.555795 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.46]   Dedup: 1.45->[1.46-1.46]
W20260812 08:02:02.555871 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 258) took 1013 ms (client timeout 50 ms). Trace:
W20260812 08:02:02.555938 31937 rpcz_store.cc:269] 0812 08:02:01.542285 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.542412 (+   127us) service_pool.cc:224] Handling call
0812 08:02:02.555191 (+1012779us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:02.555646 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.555652 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.555663 (+    11us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.555670 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.555687 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.555811 (+   124us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:02.555852 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2328090880}
W20260812 08:02:02.555421 31935 rpcz_store.cc:269] 0812 08:02:01.923011 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.923122 (+   111us) service_pool.cc:224] Handling call
0812 08:02:02.554432 (+631310us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:02.554787 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 44
0812 08:02:02.554791 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:02.554804 (+    13us) raft_consensus.cc:1645] Marking committed up to 44
0812 08:02:02.554808 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.554822 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:02.555221 (+   399us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:02.555264 (+    43us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1451319168}
W20260812 08:02:02.557137 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 910ms of latency in SegmentAllocator::Sync()
W20260812 08:02:02.625502 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 11: this message will repeat every 5th retry.
I20260812 08:02:03.365247 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.365247 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 241) took 949 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.365564 32087 rpcz_store.cc:269] 0812 08:02:02.415340 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.415391 (+    51us) service_pool.cc:224] Handling call
0812 08:02:02.415541 (+   150us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:02.415632 (+    91us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:02.415635 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:02.415765 (+   130us) write_op.cc:270] Start()
0812 08:02:02.415792 (+    27us) write_op.cc:276] Timestamp: P: 1786521722414421 usec, L: 0
0812 08:02:02.416114 (+   322us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:02.416151 (+    37us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:02.416158 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.416174 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:03.364941 (+948767us) raft_consensus.cc:1685] finished
0812 08:02:03.364944 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.365204 (+   260us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:02.416341 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:02.416575 (+   234us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:02.416579 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:02.416580 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:02.416783 (+   203us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:02.416792 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:02.416794 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:02.416985 (+   191us) tablet.cc:701] Row locks acquired
0812 08:02:02.416986 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":459,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":702,"thread_start_us":73,"threads_started":1}]]}
W20260812 08:02:03.365849 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 243) took 507 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.365903 32084 rpcz_store.cc:269] 0812 08:02:02.858797 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.858922 (+   125us) service_pool.cc:224] Handling call
0812 08:02:03.365170 (+506248us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.365616 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.365619 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.365632 (+    13us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.365638 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.365654 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.365806 (+   152us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:03.365833 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1163561856}
I20260812 08:02:03.365947 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
I20260812 08:02:03.366710 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.366799 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 244) took 456 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.366909 32085 rpcz_store.cc:269] 0812 08:02:02.910482 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.910648 (+   166us) service_pool.cc:224] Handling call
0812 08:02:03.365846 (+455198us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.366526 (+   680us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.366531 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.366541 (+    10us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.366546 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.366561 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.366751 (+   190us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:03.366783 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1045871872}
I20260812 08:02:03.366977 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.367058 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 245) took 127 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.367138 32090 rpcz_store.cc:269] 0812 08:02:03.239713 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.239851 (+   138us) service_pool.cc:224] Handling call
0812 08:02:03.366673 (+126822us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.366864 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.366867 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.366875 (+     8us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.366879 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.366890 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.367012 (+   122us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:03.367041 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":291176320}
W20260812 08:02:03.367301 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 242) took 900 ms (client timeout 50 ms). Trace:
I20260812 08:02:03.367411 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.367376 32089 rpcz_store.cc:269] 0812 08:02:02.466730 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.466805 (+    75us) service_pool.cc:224] Handling call
0812 08:02:03.366948 (+900143us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.367070 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.367076 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.367084 (+     8us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.367088 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.367101 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.367256 (+   155us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:03.367288 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069612416}
W20260812 08:02:03.367705 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 246) took 76 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.367766 32086 rpcz_store.cc:269] 0812 08:02:03.290842 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.291132 (+   290us) service_pool.cc:224] Handling call
0812 08:02:03.367385 (+ 76253us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.367523 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.367526 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.367532 (+     6us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.367536 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.367549 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.367658 (+   109us) spinlock_profiling.cc:243] Waited 76.1 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:03.367693 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174924672}
W20260812 08:02:03.415902 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19b80
I20260812 08:02:03.467849 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.468027 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 261) took 943 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.468384 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 264) took 276 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.468178 31939 rpcz_store.cc:269] 0812 08:02:02.524810 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.524939 (+   129us) service_pool.cc:224] Handling call
0812 08:02:02.555719 (+ 30780us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:02.555924 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:02.556172 (+   248us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:02.556293 (+   121us) write_op.cc:270] Start()
0812 08:02:02.556331 (+    38us) write_op.cc:276] Timestamp: P: 1786521722414421 usec, L: 0
0812 08:02:02.556842 (+   511us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:02.556892 (+    50us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:02.556899 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:02.556915 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:03.467637 (+910722us) raft_consensus.cc:1685] finished
0812 08:02:03.467640 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.467936 (+   296us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:03.468001 (+    65us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:02.556976 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:02.557130 (+   154us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:02.557134 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:02.557136 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:02.557476 (+   340us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:02.557486 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:02.557489 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:02.559333 (+  1844us) tablet.cc:701] Row locks acquired
0812 08:02:02.559335 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":70236544,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":542,"prepare.run_cpu_time_us":924,"prepare.run_wall_time_us":2431,"row_lock_wait_count":1,"row_lock_wait_us":1634,"thread_start_us":85,"threads_started":1}]]}
I20260812 08:02:03.468536 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.468477 31941 rpcz_store.cc:269] 0812 08:02:03.192385 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.192688 (+   303us) service_pool.cc:224] Handling call
0812 08:02:03.467777 (+275089us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:03.468069 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.468073 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.468087 (+    14us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.468093 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.468114 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.468330 (+   216us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:03.468367 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":631911296}
I20260812 08:02:03.469004 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.46]   Dedup: 1.46->[]
W20260812 08:02:03.469065 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 262) took 893 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.469123 31935 rpcz_store.cc:269] 0812 08:02:02.575266 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.575372 (+   106us) service_pool.cc:224] Handling call
0812 08:02:03.468494 (+893122us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:03.468826 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.468833 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.468844 (+    11us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.468849 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.468866 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.468996 (+   130us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:03.469048 (+    52us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2052630784}
W20260812 08:02:03.469297 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 263) took 328 ms (client timeout 50 ms). Trace:
W20260812 08:02:03.469386 31937 rpcz_store.cc:269] 0812 08:02:03.140723 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.141026 (+   303us) service_pool.cc:224] Handling call
0812 08:02:03.468969 (+327943us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:03.469123 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 45
0812 08:02:03.469126 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:03.469134 (+     8us) raft_consensus.cc:1645] Marking committed up to 45
0812 08:02:03.469137 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.469148 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:03.469255 (+   107us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:03.469285 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753522560}
I20260812 08:02:03.649176 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.46]   Dedup: 1.46->[]
I20260812 08:02:03.652833 32726 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=44, attempt_no=0}) took 22582 ms. Trace:
I20260812 08:02:03.653033 32726 rpcz_store.cc:276] 0812 08:01:41.070545 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:41.070708 (+   163us) service_pool.cc:224] Handling call
0812 08:02:03.652810 (+22582102us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:01.452414 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:01.452559 (+   145us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:01.452566 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:01.452567 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:01.452968 (+   401us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:01.452977 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:01.452982 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:02.414440 (+961458us) lock_manager.cc:598] Waited 960990us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:02.414460 (+    20us) tablet.cc:701] Row locks acquired
0812 08:02:02.414462 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:02.414507 (+    45us) write_op.cc:270] Start()
0812 08:02:02.414537 (+    30us) write_op.cc:276] Timestamp: P: 1786521722414421 usec, L: 0
0812 08:02:02.414542 (+     5us) op_driver.cc:348] REPLICATION: starting
0812 08:02:02.414736 (+   194us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:03.650455 (+1235719us) op_driver.cc:464] REPLICATION: finished
0812 08:02:03.650801 (+   346us) write_op.cc:301] APPLY: starting
0812 08:02:03.650838 (+    37us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:03.650914 (+    76us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:03.650919 (+     5us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:03.652355 (+  1436us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:03.652412 (+    57us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:03.652418 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:03.652586 (+   168us) log.cc:844] Serialized 319 byte log entry
0812 08:02:03.652675 (+    89us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:03.652715 (+    40us) write_op.cc:454] Released schema lock
0812 08:02:03.652723 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":274,"num_ops":50,"prepare.queue_time_us":20381208,"prepare.run_cpu_time_us":1443,"prepare.run_wall_time_us":962416,"raft.queue_time_us":327,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":204,"replication_time_us":1235863,"row_lock_wait_count":1,"row_lock_wait_us":960990,"thread_start_us":148,"threads_started":1}]]}
W20260812 08:02:03.653990 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 977ms of latency in SegmentAllocator::Sync()
W20260812 08:02:03.655594 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1040ms of latency in SegmentAllocator::Sync()
I20260812 08:02:03.697928 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.47]   Dedup: 1.46->[1.47-1.47]
W20260812 08:02:03.699728 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1144ms of latency in SegmentAllocator::Sync()
W20260812 08:02:03.756141 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:04.654151 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d198c0
W20260812 08:02:04.696220 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 249) took 1042 ms (client timeout 50 ms). Trace:
I20260812 08:02:04.696295 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.696353 32086 rpcz_store.cc:269] 0812 08:02:03.653944 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.654011 (+    67us) service_pool.cc:224] Handling call
0812 08:02:03.654167 (+   156us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:03.654248 (+    81us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:03.654252 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:03.654892 (+   640us) write_op.cc:270] Start()
0812 08:02:03.654925 (+    33us) write_op.cc:276] Timestamp: P: 1786521723652842 usec, L: 0
0812 08:02:03.655296 (+   371us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:03.655320 (+    24us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:03.655326 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.655339 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:04.696095 (+1040756us) raft_consensus.cc:1685] finished
0812 08:02:04.696100 (+     5us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.696192 (+    92us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:03.655222 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:03.655340 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:03.655345 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:03.655346 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:03.655688 (+   342us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:03.655700 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:03.655704 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:03.655984 (+   280us) tablet.cc:701] Row locks acquired
0812 08:02:03.655985 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":837,"prepare.run_wall_time_us":837,"thread_start_us":88,"threads_started":1}]]}
I20260812 08:02:04.696578 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.696663 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 250) took 990 ms (client timeout 50 ms). Trace:
I20260812 08:02:04.696799 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.696812 32089 rpcz_store.cc:269] 0812 08:02:03.705953 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.706142 (+   189us) service_pool.cc:224] Handling call
0812 08:02:04.696237 (+990095us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:04.696459 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.696464 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.696479 (+    15us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.696486 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.696501 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.696611 (+   110us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:04.696648 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276181248}
W20260812 08:02:04.697147 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 253) took 46 ms (client timeout 50 ms). Trace:
W20260812 08:02:04.696861 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 252) took 407 ms (client timeout 50 ms). Trace:
I20260812 08:02:04.697028 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.697276 32084 rpcz_store.cc:269] 0812 08:02:04.650632 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.650715 (+    83us) service_pool.cc:224] Handling call
0812 08:02:04.696766 (+ 46051us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:04.696922 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.696927 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.696938 (+    11us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.696944 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.696960 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.697080 (+   120us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:04.697134 (+    54us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105285504}
W20260812 08:02:04.697319 32085 rpcz_store.cc:269] 0812 08:02:04.289880 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.289994 (+   114us) service_pool.cc:224] Handling call
0812 08:02:04.696546 (+406552us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:04.696676 (+   130us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.696682 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.696689 (+     7us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.696692 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.696701 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.696822 (+   121us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:04.696848 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934559616}
W20260812 08:02:04.697665 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 251) took 459 ms (client timeout 50 ms). Trace:
W20260812 08:02:04.697733 32090 rpcz_store.cc:269] 0812 08:02:04.238443 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.238561 (+   118us) service_pool.cc:224] Handling call
0812 08:02:04.697005 (+458444us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:04.697475 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.697479 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.697488 (+     9us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.697492 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.697506 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.697617 (+   111us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:04.697654 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053925760}
I20260812 08:02:04.701045 32734 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=45, attempt_no=0}) took 22608 ms. Trace:
I20260812 08:02:04.701282 32734 rpcz_store.cc:276] 0812 08:01:42.092609 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:42.092771 (+   162us) service_pool.cc:224] Handling call
0812 08:02:04.701027 (+22608256us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:02.414856 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:02.414981 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:02.414986 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:02.414987 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:02.415426 (+   439us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:02.415433 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:02.415436 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:03.652727 (+1237291us) lock_manager.cc:598] Waited 1236954us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:03.652740 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:03.652742 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:03.652829 (+    87us) write_op.cc:270] Start()
0812 08:02:03.652854 (+    25us) write_op.cc:276] Timestamp: P: 1786521723652842 usec, L: 0
0812 08:02:03.652858 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:02:03.653039 (+   181us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:04.698087 (+1045048us) op_driver.cc:464] REPLICATION: finished
0812 08:02:04.698348 (+   261us) write_op.cc:301] APPLY: starting
0812 08:02:04.698389 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:04.698462 (+    73us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:04.698464 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:04.700135 (+  1671us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:04.700191 (+    56us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:04.700196 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:04.700341 (+   145us) log.cc:844] Serialized 319 byte log entry
0812 08:02:04.700417 (+    76us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:04.700951 (+   534us) write_op.cc:454] Released schema lock
0812 08:02:04.700960 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":199,"num_ops":50,"prepare.queue_time_us":20321463,"prepare.run_cpu_time_us":1602,"prepare.run_wall_time_us":1238310,"raft.queue_time_us":483,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":207,"replication_time_us":1045188,"row_lock_wait_count":1,"row_lock_wait_us":1236954,"spinlock_wait_cycles":88448,"thread_start_us":90,"threads_started":1}]]}
W20260812 08:02:04.701071 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 933ms of latency in SegmentAllocator::Sync()
W20260812 08:02:04.703604 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1014ms of latency in SegmentAllocator::Sync()
W20260812 08:02:04.751483 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 20 similar messages]
W20260812 08:02:04.802784 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:04.844424 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 265) took 1147 ms (client timeout 50 ms). Trace:
I20260812 08:02:04.844656 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.47]   Dedup: 1.47->[]
I20260812 08:02:04.845263 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.845579 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 266) took 1096 ms (client timeout 50 ms). Trace:
W20260812 08:02:04.845650 31935 rpcz_store.cc:269] 0812 08:02:03.748658 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.748815 (+   157us) service_pool.cc:224] Handling call
0812 08:02:04.844455 (+1095640us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:04.844936 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.844938 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.844949 (+    11us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.844952 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.844962 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.845430 (+   468us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:04.845560 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2519059328}
W20260812 08:02:04.845752 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 268) took 607 ms (client timeout 50 ms). Trace:
W20260812 08:02:04.844590 31937 rpcz_store.cc:269] 0812 08:02:03.697409 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.697602 (+   193us) service_pool.cc:224] Handling call
0812 08:02:03.697857 (+   255us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:03.698086 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:03.698571 (+   485us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:03.698809 (+   238us) write_op.cc:270] Start()
0812 08:02:03.698842 (+    33us) write_op.cc:276] Timestamp: P: 1786521723652842 usec, L: 0
0812 08:02:03.699272 (+   430us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:03.699323 (+    51us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:03.699329 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:03.699344 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:04.844264 (+1144920us) raft_consensus.cc:1685] finished
0812 08:02:04.844267 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.844381 (+   114us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:03.699160 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:03.699319 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:03.699324 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:03.699326 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:03.699665 (+   339us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:03.699675 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:03.699683 (+     8us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:03.700779 (+  1096us) tablet.cc:701] Row locks acquired
0812 08:02:03.700781 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":217,"prepare.run_cpu_time_us":1049,"prepare.run_wall_time_us":1703,"row_lock_wait_count":1,"row_lock_wait_us":740,"thread_start_us":127,"threads_started":1}]]}
W20260812 08:02:04.845788 31939 rpcz_store.cc:269] 0812 08:02:04.238239 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.238302 (+    63us) service_pool.cc:224] Handling call
0812 08:02:04.845221 (+606919us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:04.845552 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.845557 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.845567 (+    10us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.845576 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.845592 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.845717 (+   125us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:04.845741 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1394747136}
I20260812 08:02:04.845983 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.47]   Dedup: 1.47->[]
W20260812 08:02:04.846236 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 267) took 658 ms (client timeout 50 ms). Trace:
I20260812 08:02:04.846418 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.47->[1.48-1.48]
W20260812 08:02:04.846385 31941 rpcz_store.cc:269] 0812 08:02:04.187350 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.187434 (+    84us) service_pool.cc:224] Handling call
0812 08:02:04.845956 (+658522us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:04.846064 (+   108us) raft_consensus.cc:1534] Early marking committed up to index 46
0812 08:02:04.846070 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:04.846077 (+     7us) raft_consensus.cc:1645] Marking committed up to 46
0812 08:02:04.846080 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.846089 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:04.846195 (+   106us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:04.846223 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1513374976}
W20260812 08:02:04.847579 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 884ms of latency in SegmentAllocator::Sync()
W20260812 08:02:05.270233 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 16: this message will repeat every 5th retry.
W20260812 08:02:05.701887 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19600
W20260812 08:02:05.718369 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 255) took 1016 ms (client timeout 50 ms). Trace:
I20260812 08:02:05.718410 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.48]   Dedup: 1.48->[]
I20260812 08:02:05.718887 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.48]   Dedup: 1.48->[]
W20260812 08:02:05.718546 32089 rpcz_store.cc:269] 0812 08:02:04.701810 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.701975 (+   165us) service_pool.cc:224] Handling call
0812 08:02:04.702316 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:04.702414 (+    98us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:04.702418 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:04.702576 (+   158us) write_op.cc:270] Start()
0812 08:02:04.702630 (+    54us) write_op.cc:276] Timestamp: P: 1786521724700449 usec, L: 0
0812 08:02:04.703342 (+   712us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:04.703429 (+    87us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:04.703436 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.703449 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:05.718109 (+1014660us) raft_consensus.cc:1685] finished
0812 08:02:05.718112 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.718335 (+   223us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:04.703033 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:04.703179 (+   146us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:04.703183 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:04.703185 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:04.703473 (+   288us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:04.703485 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:04.703488 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:04.703792 (+   304us) tablet.cc:701] Row locks acquired
0812 08:02:04.703793 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":311,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":832,"thread_start_us":207,"threads_started":1}]]}
W20260812 08:02:05.719241 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 257) took 437 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.719291 32085 rpcz_store.cc:269] 0812 08:02:05.282119 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.282245 (+   126us) service_pool.cc:224] Handling call
0812 08:02:05.718851 (+436606us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:05.719049 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.719052 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.719060 (+     8us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.719063 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.719073 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.719203 (+   130us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:05.719228 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1003618560}
W20260812 08:02:05.718951 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 258) took 385 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.719408 32084 rpcz_store.cc:269] 0812 08:02:05.333359 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.333438 (+    79us) service_pool.cc:224] Handling call
0812 08:02:05.718366 (+384928us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:05.718707 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.718714 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.718727 (+    13us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.718734 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.718753 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.718899 (+   146us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:05.718935 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":884784512}
I20260812 08:02:05.719522 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.48]   Dedup: 1.48->[]
I20260812 08:02:05.719897 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.48]   Dedup: 1.48->[]
W20260812 08:02:05.719951 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 256) took 967 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.720000 32090 rpcz_store.cc:269] 0812 08:02:04.752650 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.752771 (+   121us) service_pool.cc:224] Handling call
0812 08:02:05.719499 (+966728us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:05.719745 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.719752 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.719762 (+    10us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.719766 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.719778 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.719908 (+   130us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:05.719939 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2222820736}
W20260812 08:02:05.724422 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1021ms of latency in SegmentAllocator::Sync()
I20260812 08:02:05.725056 32741 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=46, attempt_no=0}) took 22345 ms. Trace:
I20260812 08:02:05.725143 32741 rpcz_store.cc:276] 0812 08:01:43.379991 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:43.380101 (+   110us) service_pool.cc:224] Handling call
0812 08:02:05.725041 (+22344940us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:03.653197 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:03.653321 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:03.653325 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:03.653327 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:03.653702 (+   375us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:03.653710 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:03.653714 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:04.700455 (+1046741us) lock_manager.cc:598] Waited 1046416us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:04.700467 (+    12us) tablet.cc:701] Row locks acquired
0812 08:02:04.700470 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:02:04.700523 (+    53us) write_op.cc:270] Start()
0812 08:02:04.700545 (+    22us) write_op.cc:276] Timestamp: P: 1786521724700449 usec, L: 0
0812 08:02:04.700546 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:02:04.700719 (+   173us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:05.720932 (+1020213us) op_driver.cc:464] REPLICATION: finished
0812 08:02:05.721290 (+   358us) write_op.cc:301] APPLY: starting
0812 08:02:05.721336 (+    46us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:05.721402 (+    66us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:05.721403 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:05.723545 (+  2142us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:05.723638 (+    93us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:05.723646 (+     8us) write_op.cc:312] APPLY: finished
0812 08:02:05.723820 (+   174us) log.cc:844] Serialized 319 byte log entry
0812 08:02:05.723893 (+    73us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:05.724957 (+  1064us) write_op.cc:454] Released schema lock
0812 08:02:05.724966 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":242,"num_ops":50,"prepare.queue_time_us":20272391,"prepare.run_cpu_time_us":1394,"prepare.run_wall_time_us":1047609,"raft.queue_time_us":210,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":1020341,"row_lock_wait_count":1,"row_lock_wait_us":1046416,"thread_start_us":122,"threads_started":1}]]}
W20260812 08:02:05.726228 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 918ms of latency in SegmentAllocator::Sync()
I20260812 08:02:05.732378 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.48->[]
I20260812 08:02:05.732749 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.48->[]
W20260812 08:02:05.732802 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 269) took 984 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.732772 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 271) took 564 ms (client timeout 50 ms). Trace:
I20260812 08:02:05.733047 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.48->[]
W20260812 08:02:05.733119 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 270) took 933 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.733000 31941 rpcz_store.cc:269] 0812 08:02:05.168777 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.168935 (+   158us) service_pool.cc:224] Handling call
0812 08:02:05.732327 (+563392us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:05.732526 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.732528 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.732538 (+    10us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.732541 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.732551 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.732703 (+   152us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:05.732752 (+    49us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1295102336}
W20260812 08:02:05.732954 31942 rpcz_store.cc:269] 0812 08:02:04.748739 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.748873 (+   134us) service_pool.cc:224] Handling call
0812 08:02:04.846254 (+ 97381us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:04.846526 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:04.846822 (+   296us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:04.846960 (+   138us) write_op.cc:270] Start()
0812 08:02:04.846985 (+    25us) write_op.cc:276] Timestamp: P: 1786521724700449 usec, L: 0
0812 08:02:04.847301 (+   316us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:04.847339 (+    38us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:04.847344 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:04.847354 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:05.732186 (+884832us) raft_consensus.cc:1685] finished
0812 08:02:05.732188 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.732725 (+   537us) spinlock_profiling.cc:243] Waited 97.1 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:05.732779 (+    54us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:04.847257 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:04.847412 (+   155us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:04.847417 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:04.847418 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:04.847945 (+   527us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:04.847956 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:04.847958 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:04.848424 (+   466us) tablet.cc:701] Row locks acquired
0812 08:02:04.848426 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":223433600,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":174,"prepare.run_cpu_time_us":1902,"prepare.run_wall_time_us":1975,"spinlock_wait_cycles":762624,"thread_start_us":75,"threads_started":1}]]}
W20260812 08:02:05.733206 31936 rpcz_store.cc:269] 0812 08:02:04.799991 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.800115 (+   124us) service_pool.cc:224] Handling call
0812 08:02:05.732710 (+932595us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:05.732919 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.732926 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.732937 (+    11us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.732945 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.732958 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.733080 (+   122us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:05.733107 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2144121216}
W20260812 08:02:05.733366 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 272) took 513 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.733422 31935 rpcz_store.cc:269] 0812 08:02:05.220156 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.220245 (+    89us) service_pool.cc:224] Handling call
0812 08:02:05.733009 (+512764us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:05.733181 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.733187 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.733198 (+    11us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.733206 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.733220 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.733331 (+   111us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:05.733354 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1178649984}
I20260812 08:02:05.733520 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.48->[]
I20260812 08:02:05.733775 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.48]   Dedup: 1.48->[]
W20260812 08:02:05.734072 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 273) took 79 ms (client timeout 50 ms). Trace:
W20260812 08:02:05.734126 31937 rpcz_store.cc:269] 0812 08:02:05.654680 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.654887 (+   207us) service_pool.cc:224] Handling call
0812 08:02:05.733743 (+ 78856us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:05.733885 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 47
0812 08:02:05.733888 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:05.733898 (+    10us) raft_consensus.cc:1645] Marking committed up to 47
0812 08:02:05.733906 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.733921 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:05.734033 (+   112us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:05.734059 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":180778368}
W20260812 08:02:05.736373 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 982ms of latency in SegmentAllocator::Sync()
W20260812 08:02:05.826733 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:05.835999 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:06.644830 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 261) took 919 ms (client timeout 50 ms). Trace:
I20260812 08:02:06.644908 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.644969 32086 rpcz_store.cc:269] 0812 08:02:05.725089 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.725171 (+    82us) service_pool.cc:224] Handling call
0812 08:02:05.725342 (+   171us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:05.725446 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:05.725451 (+     5us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:05.725544 (+    93us) write_op.cc:270] Start()
0812 08:02:05.725569 (+    25us) write_op.cc:276] Timestamp: P: 1786521725723950 usec, L: 0
0812 08:02:05.725887 (+   318us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:05.725921 (+    34us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:05.725926 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.725937 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:06.644698 (+918761us) raft_consensus.cc:1685] finished
0812 08:02:06.644702 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.644802 (+   100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:05.726036 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:05.726193 (+   157us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:05.726197 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:05.726198 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:05.726544 (+   346us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:05.726555 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:05.726559 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:05.726947 (+   388us) tablet.cc:701] Row locks acquired
0812 08:02:05.726948 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":391,"prepare.run_cpu_time_us":972,"prepare.run_wall_time_us":972,"thread_start_us":104,"threads_started":1}]]}
I20260812 08:02:06.645224 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.645367 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 264) took 482 ms (client timeout 50 ms). Trace:
I20260812 08:02:06.645514 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.645848 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 263) took 534 ms (client timeout 50 ms). Trace:
W20260812 08:02:06.645478 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 262) took 869 ms (client timeout 50 ms). Trace:
W20260812 08:02:06.645439 32085 rpcz_store.cc:269] 0812 08:02:06.162712 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.162902 (+   190us) service_pool.cc:224] Handling call
0812 08:02:06.644859 (+481957us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.645089 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.645097 (+     8us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.645108 (+    11us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.645112 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.645122 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.645319 (+   197us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:06.645352 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1107878272}
W20260812 08:02:06.645931 32084 rpcz_store.cc:269] 0812 08:02:06.111474 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.111546 (+    72us) service_pool.cc:224] Handling call
0812 08:02:06.645481 (+533935us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.645644 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.645654 (+    10us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.645686 (+    32us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.645692 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.645709 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.645812 (+   103us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:06.645836 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1227326208}
W20260812 08:02:06.646023 32090 rpcz_store.cc:269] 0812 08:02:05.776443 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.776562 (+   119us) service_pool.cc:224] Handling call
0812 08:02:06.645197 (+868635us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.645329 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.645331 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.645338 (+     7us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.645341 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.645350 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.645453 (+   103us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:06.645470 (+    17us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997243264}
I20260812 08:02:06.652989 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.719187 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 275) took 984 ms (client timeout 50 ms). Trace:
I20260812 08:02:06.719188 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
I20260812 08:02:06.719769 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.719889 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 276) took 934 ms (client timeout 50 ms). Trace:
W20260812 08:02:06.719386 31937 rpcz_store.cc:269] 0812 08:02:05.735137 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.735223 (+    86us) service_pool.cc:224] Handling call
0812 08:02:05.735392 (+   169us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:05.735489 (+    97us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:05.735680 (+   191us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:05.735815 (+   135us) write_op.cc:270] Start()
0812 08:02:05.735841 (+    26us) write_op.cc:276] Timestamp: P: 1786521725723950 usec, L: 0
0812 08:02:05.736125 (+   284us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:05.736161 (+    36us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:05.736166 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:05.736177 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:06.718926 (+982749us) raft_consensus.cc:1685] finished
0812 08:02:06.718930 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.719136 (+   206us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:05.736048 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:05.736166 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:05.736171 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:05.736172 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:05.736572 (+   400us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:05.736583 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:05.736585 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:05.737848 (+  1263us) tablet.cc:701] Row locks acquired
0812 08:02:05.737851 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":1044,"prepare.run_wall_time_us":1868,"row_lock_wait_count":1,"row_lock_wait_us":905,"thread_start_us":78,"threads_started":1}]]}
W20260812 08:02:06.719944 31939 rpcz_store.cc:269] 0812 08:02:05.785915 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.786072 (+   157us) service_pool.cc:224] Handling call
0812 08:02:06.719137 (+933065us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.719622 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.719625 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.719637 (+    12us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.719640 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.719651 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.719800 (+   149us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:06.719870 (+    70us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2145069312}
W20260812 08:02:06.720048 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 277) took 487 ms (client timeout 50 ms). Trace:
W20260812 08:02:06.720096 31935 rpcz_store.cc:269] 0812 08:02:06.232302 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.232404 (+   102us) service_pool.cc:224] Handling call
0812 08:02:06.719737 (+487333us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.719881 (+   144us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.719884 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.719890 (+     6us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.719894 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.719907 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.720018 (+   111us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:06.720039 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1120199424}
I20260812 08:02:06.720353 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.49]   Dedup: 1.49->[]
W20260812 08:02:06.720618 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 278) took 437 ms (client timeout 50 ms). Trace:
W20260812 08:02:06.720698 31936 rpcz_store.cc:269] 0812 08:02:06.283327 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.283462 (+   135us) service_pool.cc:224] Handling call
0812 08:02:06.720325 (+436863us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.720461 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 48
0812 08:02:06.720465 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:06.720474 (+     9us) raft_consensus.cc:1645] Marking committed up to 48
0812 08:02:06.720479 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.720489 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:06.720579 (+    90us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:06.720604 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1004224256}
W20260812 08:02:06.725111 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19340
I20260812 08:02:06.748792 32746 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=47, attempt_no=0}) took 22155 ms. Trace:
I20260812 08:02:06.749006 32746 rpcz_store.cc:276] 0812 08:01:44.593543 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:44.593628 (+    85us) service_pool.cc:224] Handling call
0812 08:02:06.748768 (+22155140us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:04.700833 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:04.700956 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:04.700959 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:04.700961 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:04.701269 (+   308us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:04.701277 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:04.701280 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:05.723930 (+1022650us) lock_manager.cc:598] Waited 1022388us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:05.723943 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:05.723945 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:05.723992 (+    47us) write_op.cc:270] Start()
0812 08:02:05.724012 (+    20us) write_op.cc:276] Timestamp: P: 1786521725723950 usec, L: 0
0812 08:02:05.724014 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:05.724152 (+   138us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:06.746050 (+1021898us) op_driver.cc:464] REPLICATION: finished
0812 08:02:06.746429 (+   379us) write_op.cc:301] APPLY: starting
0812 08:02:06.746468 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:06.746527 (+    59us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:06.746528 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:06.748323 (+  1795us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:06.748407 (+    84us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:06.748412 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:06.748551 (+   139us) log.cc:844] Serialized 319 byte log entry
0812 08:02:06.748629 (+    78us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:06.748670 (+    41us) write_op.cc:454] Released schema lock
0812 08:02:06.748680 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":309,"num_ops":50,"prepare.queue_time_us":20106844,"prepare.run_cpu_time_us":1290,"prepare.run_wall_time_us":1023385,"raft.queue_time_us":429,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":197,"replication_time_us":1021985,"row_lock_wait_count":1,"row_lock_wait_us":1022388,"thread_start_us":125,"threads_started":1}]]}
W20260812 08:02:06.749308 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1057ms of latency in SegmentAllocator::Sync()
W20260812 08:02:06.750984 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 919ms of latency in SegmentAllocator::Sync()
I20260812 08:02:06.754312 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.50]   Dedup: 1.49->[1.50-1.50]
W20260812 08:02:06.755475 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 929ms of latency in SegmentAllocator::Sync()
W20260812 08:02:06.851864 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:07.670987 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 267) took 921 ms (client timeout 50 ms). Trace:
I20260812 08:02:07.671026 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.671104 32085 rpcz_store.cc:269] 0812 08:02:06.749928 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.749992 (+    64us) service_pool.cc:224] Handling call
0812 08:02:06.750136 (+   144us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:06.750218 (+    82us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:06.750220 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:06.750293 (+    73us) write_op.cc:270] Start()
0812 08:02:06.750315 (+    22us) write_op.cc:276] Timestamp: P: 1786521726748809 usec, L: 0
0812 08:02:06.750755 (+   440us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:06.750817 (+    62us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:06.750823 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.750836 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:07.670861 (+920025us) raft_consensus.cc:1685] finished
0812 08:02:07.670864 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.670961 (+    97us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:06.750666 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:06.750778 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:06.750782 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:06.750784 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:06.751216 (+   432us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:06.751229 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:06.751233 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:06.751530 (+   297us) tablet.cc:701] Row locks acquired
0812 08:02:06.751532 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":273,"prepare.run_cpu_time_us":941,"prepare.run_wall_time_us":940,"thread_start_us":127,"threads_started":1}]]}
I20260812 08:02:07.671357 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.671442 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 268) took 869 ms (client timeout 50 ms). Trace:
I20260812 08:02:07.671564 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.671499 32090 rpcz_store.cc:269] 0812 08:02:06.801889 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.802011 (+   122us) service_pool.cc:224] Handling call
0812 08:02:07.670980 (+868969us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:07.671234 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.671242 (+     8us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.671252 (+    10us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.671255 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.671265 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.671394 (+   129us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:07.671427 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997797632}
W20260812 08:02:07.671597 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 269) took 544 ms (client timeout 50 ms). Trace:
W20260812 08:02:07.671975 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 270) took 493 ms (client timeout 50 ms). Trace:
W20260812 08:02:07.672055 32086 rpcz_store.cc:269] 0812 08:02:07.178504 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.178636 (+   132us) service_pool.cc:224] Handling call
0812 08:02:07.671529 (+492893us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:07.671780 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.671785 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.671794 (+     9us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.671801 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.671814 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.671939 (+   125us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:07.671963 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133081856}
W20260812 08:02:07.671880 32084 rpcz_store.cc:269] 0812 08:02:07.127008 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.127123 (+   115us) service_pool.cc:224] Handling call
0812 08:02:07.671327 (+544204us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:07.671449 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.671461 (+    12us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.671468 (+     7us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.671471 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.671480 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.671565 (+    85us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:07.671586 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1251177856}
W20260812 08:02:07.685017 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 279) took 931 ms (client timeout 50 ms). Trace:
I20260812 08:02:07.685034 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.50]   Dedup: 1.50->[]
I20260812 08:02:07.685441 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.685165 31936 rpcz_store.cc:269] 0812 08:02:06.753974 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.754050 (+    76us) service_pool.cc:224] Handling call
0812 08:02:06.754224 (+   174us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:06.754481 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:06.754736 (+   255us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:06.754866 (+   130us) write_op.cc:270] Start()
0812 08:02:06.754891 (+    25us) write_op.cc:276] Timestamp: P: 1786521726748809 usec, L: 0
0812 08:02:06.755227 (+   336us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:06.755269 (+    42us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:06.755275 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:06.755288 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:07.684905 (+929617us) raft_consensus.cc:1685] finished
0812 08:02:07.684907 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.684992 (+    85us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:06.755149 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:06.755271 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:06.755275 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:06.755277 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:06.755501 (+   224us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:06.755509 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:06.755511 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:06.756821 (+  1310us) tablet.cc:701] Row locks acquired
0812 08:02:06.756825 (+     4us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":190,"prepare.run_cpu_time_us":896,"prepare.run_wall_time_us":1758,"row_lock_wait_count":1,"row_lock_wait_us":969,"thread_start_us":87,"threads_started":1}]]}
W20260812 08:02:07.685523 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 281) took 393 ms (client timeout 50 ms). Trace:
W20260812 08:02:07.685628 31935 rpcz_store.cc:269] 0812 08:02:07.292318 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.292463 (+   145us) service_pool.cc:224] Handling call
0812 08:02:07.684987 (+392524us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:07.685323 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.685326 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.685337 (+    11us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.685340 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.685351 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.685480 (+   129us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:07.685510 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902139776}
I20260812 08:02:07.685778 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.685828 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 280) took 880 ms (client timeout 50 ms). Trace:
W20260812 08:02:07.686058 31939 rpcz_store.cc:269] 0812 08:02:06.805264 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.805382 (+   118us) service_pool.cc:224] Handling call
0812 08:02:07.685412 (+880030us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:07.685560 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.685563 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.685572 (+     9us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.685577 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.685586 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.685699 (+   113us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:07.685731 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2023212288}
W20260812 08:02:07.686228 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 282) took 342 ms (client timeout 50 ms). Trace:
W20260812 08:02:07.686303 31937 rpcz_store.cc:269] 0812 08:02:07.343838 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.343959 (+   121us) service_pool.cc:224] Handling call
0812 08:02:07.685746 (+341787us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:07.686060 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 49
0812 08:02:07.686063 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:07.686074 (+    11us) raft_consensus.cc:1645] Marking committed up to 49
0812 08:02:07.686080 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.686092 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:07.686193 (+   101us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:07.686217 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":785562880}
I20260812 08:02:07.707381 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.50]   Dedup: 1.50->[]
W20260812 08:02:07.750226 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19080
I20260812 08:02:07.810690 32758 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=48, attempt_no=0}) took 22104 ms. Trace:
W20260812 08:02:07.811002 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1010ms of latency in SegmentAllocator::Sync()
I20260812 08:02:07.810913 32758 rpcz_store.cc:276] 0812 08:01:45.706733 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:45.706886 (+   153us) service_pool.cc:224] Handling call
0812 08:02:07.810666 (+22103780us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:05.724246 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:05.724358 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:05.724363 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:05.724364 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:05.724652 (+   288us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:05.724660 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:05.724663 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:06.748805 (+1024142us) lock_manager.cc:598] Waited 1023879us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:06.748820 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:06.748824 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:02:06.748869 (+    45us) write_op.cc:270] Start()
0812 08:02:06.748916 (+    47us) write_op.cc:276] Timestamp: P: 1786521726748809 usec, L: 0
0812 08:02:06.748919 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:06.749077 (+   158us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:07.807047 (+1057970us) op_driver.cc:464] REPLICATION: finished
0812 08:02:07.807655 (+   608us) write_op.cc:301] APPLY: starting
0812 08:02:07.807693 (+    38us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:07.807754 (+    61us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:07.807755 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:07.809099 (+  1344us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:07.809154 (+    55us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:07.809160 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:07.809311 (+   151us) log.cc:844] Serialized 319 byte log entry
0812 08:02:07.809390 (+    79us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:07.810547 (+  1157us) write_op.cc:454] Released schema lock
0812 08:02:07.810555 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":544,"num_ops":50,"prepare.queue_time_us":20016722,"prepare.run_cpu_time_us":1306,"prepare.run_wall_time_us":1024942,"raft.queue_time_us":766,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":228,"replication_time_us":1058079,"row_lock_wait_count":1,"row_lock_wait_us":1023879,"thread_start_us":362,"threads_started":1}]]}
W20260812 08:02:07.811569 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 867ms of latency in SegmentAllocator::Sync()
I20260812 08:02:07.854569 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.51]   Dedup: 1.50->[1.51-1.51]
W20260812 08:02:07.855850 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1212ms of latency in SegmentAllocator::Sync()
W20260812 08:02:07.913221 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:08.409559 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:02:08.679198 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 273) took 868 ms (client timeout 50 ms). Trace:
I20260812 08:02:08.679230 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.51]   Dedup: 1.51->[]
I20260812 08:02:08.679878 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.51]   Dedup: 1.51->[]
W20260812 08:02:08.679970 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 274) took 817 ms (client timeout 50 ms). Trace:
W20260812 08:02:08.679576 32084 rpcz_store.cc:269] 0812 08:02:07.810520 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.810565 (+    45us) service_pool.cc:224] Handling call
0812 08:02:07.810743 (+   178us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:07.810830 (+    87us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:07.810832 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:07.810969 (+   137us) write_op.cc:270] Start()
0812 08:02:07.810997 (+    28us) write_op.cc:276] Timestamp: P: 1786521727809451 usec, L: 0
0812 08:02:07.811331 (+   334us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:07.811364 (+    33us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:07.811370 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.811381 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:08.679067 (+867686us) raft_consensus.cc:1685] finished
0812 08:02:08.679070 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:08.679166 (+    96us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:07.811254 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:07.811365 (+   111us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:07.811368 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:07.811370 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:07.811700 (+   330us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:07.811725 (+    25us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:07.811732 (+     7us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:07.812221 (+   489us) tablet.cc:701] Row locks acquired
0812 08:02:07.812223 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":171,"prepare.run_cpu_time_us":1046,"prepare.run_wall_time_us":1046,"thread_start_us":74,"threads_started":1}]]}
I20260812 08:02:08.680174 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.51]   Dedup: 1.51->[]
W20260812 08:02:08.680099 32086 rpcz_store.cc:269] 0812 08:02:07.862887 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.863010 (+   123us) service_pool.cc:224] Handling call
0812 08:02:08.679181 (+816171us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:08.679724 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:08.679727 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:08.679739 (+    12us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:08.679742 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:08.679753 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:08.679911 (+   158us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:08.679952 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1876573568}
W20260812 08:02:08.680555 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 276) took 310 ms (client timeout 50 ms). Trace:
W20260812 08:02:08.680614 32085 rpcz_store.cc:269] 0812 08:02:08.370154 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.370245 (+    91us) service_pool.cc:224] Handling call
0812 08:02:08.680136 (+309891us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:08.680371 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:08.680374 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:08.680383 (+     9us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:08.680391 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:08.680405 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:08.680515 (+   110us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:08.680543 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":712225536}
W20260812 08:02:08.680192 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 275) took 361 ms (client timeout 50 ms). Trace:
W20260812 08:02:08.681180 32090 rpcz_store.cc:269] 0812 08:02:08.319116 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.319200 (+    84us) service_pool.cc:224] Handling call
0812 08:02:08.679844 (+360644us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:08.680001 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:08.680004 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:08.680012 (+     8us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:08.680016 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:08.680031 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:08.680155 (+   124us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:08.680179 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":828793216}
I20260812 08:02:08.701072 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.51]   Dedup: 1.51->[]
W20260812 08:02:08.810768 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d182c0
I20260812 08:02:08.825670   301 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=49, attempt_no=0}) took 22099 ms. Trace:
I20260812 08:02:08.825853   301 rpcz_store.cc:276] 0812 08:01:46.726160 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:46.726231 (+    71us) service_pool.cc:224] Handling call
0812 08:02:08.825649 (+22099418us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:06.749218 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:06.749343 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:06.749350 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:06.749352 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:06.749749 (+   397us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:06.749757 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:06.749761 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:07.809429 (+1059668us) lock_manager.cc:598] Waited 1059380us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:07.809442 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:07.809443 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:02:07.809497 (+    54us) write_op.cc:270] Start()
0812 08:02:07.809518 (+    21us) write_op.cc:276] Timestamp: P: 1786521727809451 usec, L: 0
0812 08:02:07.809520 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:07.809684 (+   164us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:08.821680 (+1011996us) op_driver.cc:464] REPLICATION: finished
0812 08:02:08.822118 (+   438us) write_op.cc:301] APPLY: starting
0812 08:02:08.822159 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:08.822216 (+    57us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:08.822218 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:08.825263 (+  3045us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:08.825324 (+    61us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:08.825329 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:08.825457 (+   128us) log.cc:844] Serialized 319 byte log entry
0812 08:02:08.825529 (+    72us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:08.825566 (+    37us) write_op.cc:454] Released schema lock
0812 08:02:08.825575 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":300,"num_ops":50,"prepare.queue_time_us":20022461,"prepare.run_cpu_time_us":1392,"prepare.run_wall_time_us":1060545,"raft.queue_time_us":308,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":8,"replication_time_us":1012105,"row_lock_wait_count":1,"row_lock_wait_us":1059380,"thread_start_us":194,"threads_started":1}]]}
W20260812 08:02:08.826874 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1057ms of latency in SegmentAllocator::Sync()
W20260812 08:02:08.828730 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1050ms of latency in SegmentAllocator::Sync()
W20260812 08:02:08.929713 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:09.068521 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 283) took 1214 ms (client timeout 50 ms). Trace:
I20260812 08:02:09.068557 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.51]   Dedup: 1.51->[]
W20260812 08:02:09.068763 31937 rpcz_store.cc:269] 0812 08:02:07.854213 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.854300 (+    87us) service_pool.cc:224] Handling call
0812 08:02:07.854496 (+   196us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:07.854819 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:07.855038 (+   219us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:07.855168 (+   130us) write_op.cc:270] Start()
0812 08:02:07.855193 (+    25us) write_op.cc:276] Timestamp: P: 1786521727809451 usec, L: 0
0812 08:02:07.855521 (+   328us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:07.855562 (+    41us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:07.855568 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:07.855583 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:09.068369 (+1212786us) raft_consensus.cc:1685] finished
0812 08:02:09.068372 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.068487 (+   115us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:07.855425 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:07.855549 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:07.855554 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:07.855555 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:07.855842 (+   287us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:07.855852 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:07.855854 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:07.856935 (+  1081us) tablet.cc:701] Row locks acquired
0812 08:02:07.856937 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":150,"prepare.run_cpu_time_us":1054,"prepare.run_wall_time_us":1905,"row_lock_wait_count":1,"row_lock_wait_us":799,"spinlock_wait_cycles":727936,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:02:09.069063 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.51]   Dedup: 1.51->[]
W20260812 08:02:09.069151 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 286) took 709 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.069234 31936 rpcz_store.cc:269] 0812 08:02:08.359420 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.359557 (+   137us) service_pool.cc:224] Handling call
0812 08:02:09.068507 (+708950us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:09.068895 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:09.068900 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.068918 (+    18us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:09.068921 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.068934 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.069088 (+   154us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.069129 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629916416}
I20260812 08:02:09.069365 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.51]   Dedup: 1.51->[]
I20260812 08:02:09.069655 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.51->[1.52-1.52]
W20260812 08:02:09.069703 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 284) took 1164 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.069383 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 285) took 761 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.069797 31935 rpcz_store.cc:269] 0812 08:02:08.308071 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.308150 (+    79us) service_pool.cc:224] Handling call
0812 08:02:09.069017 (+760867us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:09.069172 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:09.069175 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.069183 (+     8us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:09.069188 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.069202 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.069341 (+   139us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.069369 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1749037312}
W20260812 08:02:09.069783 31939 rpcz_store.cc:269] 0812 08:02:07.905541 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.905744 (+   203us) service_pool.cc:224] Handling call
0812 08:02:09.069342 (+1163598us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:09.069484 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 50
0812 08:02:09.069490 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.069501 (+    11us) raft_consensus.cc:1645] Marking committed up to 50
0812 08:02:09.069509 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.069523 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.069654 (+   131us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.069688 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2675529088}
W20260812 08:02:09.070816 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 854ms of latency in SegmentAllocator::Sync()
W20260812 08:02:09.827111 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1080
W20260812 08:02:09.879571 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 279) took 1052 ms (client timeout 50 ms). Trace:
I20260812 08:02:09.879817 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.879684 32090 rpcz_store.cc:269] 0812 08:02:08.827489 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.827725 (+   236us) service_pool.cc:224] Handling call
0812 08:02:08.827946 (+   221us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:08.828032 (+    86us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:08.828035 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:08.828170 (+   135us) write_op.cc:270] Start()
0812 08:02:08.828197 (+    27us) write_op.cc:276] Timestamp: P: 1786521728825781 usec, L: 0
0812 08:02:08.828523 (+   326us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:08.828572 (+    49us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:08.828578 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:08.828591 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:09.879439 (+1050848us) raft_consensus.cc:1685] finished
0812 08:02:09.879442 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.879543 (+   101us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:08.828494 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:08.828615 (+   121us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:08.828619 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:08.828621 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:08.828954 (+   333us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:08.828967 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:08.828971 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:08.829588 (+   617us) tablet.cc:701] Row locks acquired
0812 08:02:08.829591 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":211,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":1174,"thread_start_us":97,"threads_started":1}]]}
W20260812 08:02:09.880295 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 280) took 1000 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.880376 32085 rpcz_store.cc:269] 0812 08:02:08.879543 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.879870 (+   327us) service_pool.cc:224] Handling call
0812 08:02:09.879749 (+999879us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:09.880062 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.880065 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.880077 (+    12us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.880081 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.880093 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.880238 (+   145us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:09.880279 (+    41us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2298603392}
I20260812 08:02:09.880467 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.52]   Dedup: 1.52->[]
I20260812 08:02:09.880825 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.880870 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 281) took 454 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.880970 32086 rpcz_store.cc:269] 0812 08:02:09.426701 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.426783 (+    82us) service_pool.cc:224] Handling call
0812 08:02:09.880224 (+453441us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:09.880577 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.880581 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.880591 (+    10us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.880599 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.880615 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.880822 (+   207us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:09.880856 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1042442112}
W20260812 08:02:09.881130 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 282) took 403 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.881385 32084 rpcz_store.cc:269] 0812 08:02:09.477698 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.477814 (+   116us) service_pool.cc:224] Handling call
0812 08:02:09.880791 (+402977us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:09.880938 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.880940 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.880946 (+     6us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.880949 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.880956 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.881087 (+   131us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:09.881113 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":926286080}
I20260812 08:02:09.905373 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.52]   Dedup: 1.52->[]
I20260812 08:02:09.909425   313 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=50, attempt_no=0}) took 22267 ms. Trace:
I20260812 08:02:09.909646   313 rpcz_store.cc:276] 0812 08:01:47.642019 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:47.642254 (+   235us) service_pool.cc:224] Handling call
0812 08:02:09.909396 (+22267142us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:07.809790 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:07.809913 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:07.809919 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:07.809921 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:07.810239 (+   318us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:07.810249 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:07.810253 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:08.825657 (+1015404us) lock_manager.cc:598] Waited 1015118us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:08.825673 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:08.825675 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:08.825851 (+   176us) write_op.cc:270] Start()
0812 08:02:08.825884 (+    33us) write_op.cc:276] Timestamp: P: 1786521728825781 usec, L: 0
0812 08:02:08.825886 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:08.826082 (+   196us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:09.906692 (+1080610us) op_driver.cc:464] REPLICATION: finished
0812 08:02:09.906953 (+   261us) write_op.cc:301] APPLY: starting
0812 08:02:09.906996 (+    43us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:09.907064 (+    68us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:09.907065 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:09.908762 (+  1697us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:09.909007 (+   245us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:09.909014 (+     7us) write_op.cc:312] APPLY: finished
0812 08:02:09.909195 (+   181us) log.cc:844] Serialized 319 byte log entry
0812 08:02:09.909275 (+    80us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:09.909303 (+    28us) write_op.cc:454] Released schema lock
0812 08:02:09.909311 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"num_ops":50,"prepare.queue_time_us":20166760,"prepare.run_cpu_time_us":1647,"prepare.run_wall_time_us":1016376,"raft.queue_time_us":1740,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":353,"replication_time_us":1080750,"row_lock_wait_count":1,"row_lock_wait_us":1015118,"thread_start_us":113,"threads_started":1}]]}
W20260812 08:02:09.910058 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 723ms of latency in SegmentAllocator::Sync()
W20260812 08:02:09.911638 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1016ms of latency in SegmentAllocator::Sync()
W20260812 08:02:09.915441 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 20 similar messages]
I20260812 08:02:09.925464 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.925742 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 287) took 1046 ms (client timeout 50 ms). Trace:
I20260812 08:02:09.925886 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.925863 31942 rpcz_store.cc:269] 0812 08:02:08.879679 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.879870 (+   191us) service_pool.cc:224] Handling call
0812 08:02:09.069578 (+189708us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:09.069794 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.070060 (+   266us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:09.070171 (+   111us) write_op.cc:270] Start()
0812 08:02:09.070207 (+    36us) write_op.cc:276] Timestamp: P: 1786521728825781 usec, L: 0
0812 08:02:09.070524 (+   317us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:09.070565 (+    41us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.070571 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.070609 (+    38us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:09.925346 (+854737us) raft_consensus.cc:1685] finished
0812 08:02:09.925349 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.925591 (+   242us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.925708 (+   117us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:09.070439 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:09.070573 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:09.070605 (+    32us) write_op.cc:435] Acquired schema lock
0812 08:02:09.070607 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:09.070910 (+   303us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:09.070926 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:09.070931 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:09.071945 (+  1014us) tablet.cc:701] Row locks acquired
0812 08:02:09.071947 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":435790080,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":1573,"row_lock_wait_count":1,"row_lock_wait_us":795,"thread_start_us":77,"threads_started":1}]]}
I20260812 08:02:09.926139 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.926198 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 290) took 615 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.926247 31939 rpcz_store.cc:269] 0812 08:02:09.310940 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.311062 (+   122us) service_pool.cc:224] Handling call
0812 08:02:09.925847 (+614785us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:09.925999 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.926002 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.926011 (+     9us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.926015 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.926027 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.926156 (+   129us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.926182 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1413262208}
W20260812 08:02:09.926453 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 288) took 995 ms (client timeout 50 ms). Trace:
I20260812 08:02:09.926553 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.926867 31941 rpcz_store.cc:269] 0812 08:02:08.931254 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.931429 (+   175us) service_pool.cc:224] Handling call
0812 08:02:09.926108 (+994679us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:09.926255 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.926258 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.926265 (+     7us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.926269 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.926291 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.926412 (+   121us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.926441 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2286974848}
W20260812 08:02:09.927291 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 291) took 62 ms (client timeout 50 ms). Trace:
I20260812 08:02:09.927385 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.52]   Dedup: 1.52->[]
W20260812 08:02:09.927359 31936 rpcz_store.cc:269] 0812 08:02:09.865071 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.865297 (+   226us) service_pool.cc:224] Handling call
0812 08:02:09.926522 (+ 61225us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:09.927069 (+   547us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.927076 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.927087 (+    11us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.927095 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.927110 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.927244 (+   134us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.927274 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140186496}
W20260812 08:02:09.927675 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 289) took 668 ms (client timeout 50 ms). Trace:
W20260812 08:02:09.927733 31935 rpcz_store.cc:269] 0812 08:02:09.259689 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.259836 (+   147us) service_pool.cc:224] Handling call
0812 08:02:09.927353 (+667517us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:09.927507 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 51
0812 08:02:09.927510 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:09.927518 (+     8us) raft_consensus.cc:1645] Marking committed up to 51
0812 08:02:09.927521 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.927531 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:09.927641 (+   110us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:09.927665 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1534645504}
W20260812 08:02:09.928865 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 878ms of latency in SegmentAllocator::Sync()
W20260812 08:02:10.012494 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:10.028817 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:10.807572 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 293) took 880 ms (client timeout 50 ms). Trace:
I20260812 08:02:10.807749 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
I20260812 08:02:10.808235 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
W20260812 08:02:10.807934 31941 rpcz_store.cc:269] 0812 08:02:09.927536 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.927616 (+    80us) service_pool.cc:224] Handling call
0812 08:02:09.927775 (+   159us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:09.927851 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:09.928076 (+   225us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:09.928224 (+   148us) write_op.cc:270] Start()
0812 08:02:09.928261 (+    37us) write_op.cc:276] Timestamp: P: 1786521729909381 usec, L: 0
0812 08:02:09.928588 (+   327us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:09.928623 (+    35us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:09.928629 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.928640 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:10.807439 (+878799us) raft_consensus.cc:1685] finished
0812 08:02:10.807441 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.807544 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:09.928533 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:09.928891 (+   358us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:09.928897 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:09.928899 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:09.929336 (+   437us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:09.929350 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:09.929353 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:09.930380 (+  1027us) tablet.cc:701] Row locks acquired
0812 08:02:09.930382 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":174,"prepare.run_cpu_time_us":1601,"prepare.run_wall_time_us":1928,"row_lock_wait_count":1,"row_lock_wait_us":589,"thread_start_us":75,"threads_started":1}]]}
W20260812 08:02:10.808329 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 294) took 829 ms (client timeout 50 ms). Trace:
W20260812 08:02:10.808377 31935 rpcz_store.cc:269] 0812 08:02:09.978811 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.978957 (+   146us) service_pool.cc:224] Handling call
0812 08:02:10.807570 (+828613us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.808091 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.808097 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.808113 (+    16us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.808120 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.808134 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.808279 (+   145us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:10.808316 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1905126144}
W20260812 08:02:10.808508 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 296) took 399 ms (client timeout 50 ms). Trace:
I20260812 08:02:10.808569 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
W20260812 08:02:10.808545 31939 rpcz_store.cc:269] 0812 08:02:10.409067 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.409269 (+   202us) service_pool.cc:224] Handling call
0812 08:02:10.808206 (+398937us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.808342 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.808347 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.808355 (+     8us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.808358 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.808367 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.808478 (+   111us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:10.808497 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916412288}
I20260812 08:02:10.808964 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
W20260812 08:02:10.809027 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 295) took 451 ms (client timeout 50 ms). Trace:
W20260812 08:02:10.809100 31936 rpcz_store.cc:269] 0812 08:02:10.357833 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.357965 (+   132us) service_pool.cc:224] Handling call
0812 08:02:10.808538 (+450573us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.808811 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.808817 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.808829 (+    12us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.808835 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.808851 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.808985 (+   134us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:10.809015 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035805952}
I20260812 08:02:10.812984   318 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=51, attempt_no=0}) took 22031 ms. Trace:
I20260812 08:02:10.813114   318 rpcz_store.cc:276] 0812 08:01:48.781457 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:48.781596 (+   139us) service_pool.cc:224] Handling call
0812 08:02:10.812965 (+22031369us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:08.826202 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:08.826339 (+   137us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:08.826343 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:08.826345 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:08.826688 (+   343us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:08.826696 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:08.826700 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:09.909379 (+1082679us) lock_manager.cc:598] Waited 1082389us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:09.909391 (+    12us) tablet.cc:701] Row locks acquired
0812 08:02:09.909393 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:09.909443 (+    50us) write_op.cc:270] Start()
0812 08:02:09.909483 (+    40us) write_op.cc:276] Timestamp: P: 1786521729909381 usec, L: 0
0812 08:02:09.909487 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:02:09.909773 (+   286us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:10.810549 (+900776us) op_driver.cc:464] REPLICATION: finished
0812 08:02:10.810883 (+   334us) write_op.cc:301] APPLY: starting
0812 08:02:10.810940 (+    57us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:10.811013 (+    73us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:10.811015 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:10.812523 (+  1508us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:10.812579 (+    56us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:10.812587 (+     8us) write_op.cc:312] APPLY: finished
0812 08:02:10.812749 (+   162us) log.cc:844] Serialized 319 byte log entry
0812 08:02:10.812829 (+    80us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:10.812871 (+    42us) write_op.cc:454] Released schema lock
0812 08:02:10.812880 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":279,"num_ops":50,"prepare.queue_time_us":20043929,"prepare.run_cpu_time_us":1509,"prepare.run_wall_time_us":1083667,"raft.queue_time_us":62,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":218,"replication_time_us":901016,"row_lock_wait_count":1,"row_lock_wait_us":1082389,"thread_start_us":149,"threads_started":1}]]}
W20260812 08:02:10.813433 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 981ms of latency in SegmentAllocator::Sync()
W20260812 08:02:10.815035 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 972ms of latency in SegmentAllocator::Sync()
W20260812 08:02:10.916018 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:10.928361 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 285) took 1017 ms (client timeout 50 ms). Trace:
I20260812 08:02:10.928417 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
I20260812 08:02:10.929167 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
W20260812 08:02:10.928725 32084 rpcz_store.cc:269] 0812 08:02:09.910463 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.910534 (+    71us) service_pool.cc:224] Handling call
0812 08:02:09.910757 (+   223us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:09.910844 (+    87us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:09.910848 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:09.910975 (+   127us) write_op.cc:270] Start()
0812 08:02:09.911002 (+    27us) write_op.cc:276] Timestamp: P: 1786521729909381 usec, L: 0
0812 08:02:09.911347 (+   345us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:09.911393 (+    46us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:09.911399 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:09.911413 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:10.928220 (+1016807us) raft_consensus.cc:1685] finished
0812 08:02:10.928224 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.928327 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:09.911347 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:09.911496 (+   149us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:09.911500 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:09.911502 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:09.911896 (+   394us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:09.911906 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:09.911908 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:09.912135 (+   227us) tablet.cc:701] Row locks acquired
0812 08:02:09.912137 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":253,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":867,"thread_start_us":94,"threads_started":1}]]}
W20260812 08:02:10.929247 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 286) took 967 ms (client timeout 50 ms). Trace:
W20260812 08:02:10.929306 32086 rpcz_store.cc:269] 0812 08:02:09.962339 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.962503 (+   164us) service_pool.cc:224] Handling call
0812 08:02:10.928370 (+965867us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.928994 (+   624us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.928996 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.929007 (+    11us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.929009 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.929043 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.929154 (+   111us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:10.929232 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220673920}
I20260812 08:02:10.929394 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53]   Dedup: 1.53->[]
W20260812 08:02:10.929503 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 287) took 412 ms (client timeout 50 ms). Trace:
W20260812 08:02:10.929632 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 288) took 361 ms (client timeout 50 ms). Trace:
W20260812 08:02:10.929708 32090 rpcz_store.cc:269] 0812 08:02:10.567824 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.568140 (+   316us) service_pool.cc:224] Handling call
0812 08:02:10.929369 (+361229us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.929471 (+   102us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.929474 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.929480 (+     6us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.929483 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.929493 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.929592 (+    99us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:10.929620 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830285696}
W20260812 08:02:10.929575 32085 rpcz_store.cc:269] 0812 08:02:10.516739 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.516939 (+   200us) service_pool.cc:224] Handling call
0812 08:02:10.929124 (+412185us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.929292 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 52
0812 08:02:10.929295 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:10.929303 (+     8us) raft_consensus.cc:1645] Marking committed up to 52
0812 08:02:10.929307 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.929318 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:10.929460 (+   142us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:10.929488 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947467008}
I20260812 08:02:11.134569 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.54]   Dedup: 1.53->[1.54-1.54]
W20260812 08:02:11.135951 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 880ms of latency in SegmentAllocator::Sync()
W20260812 08:02:11.787556 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 299) took 973 ms (client timeout 50 ms). Trace:
I20260812 08:02:11.787556 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.54]   Dedup: 1.54->[]
W20260812 08:02:11.787837 31942 rpcz_store.cc:269] 0812 08:02:10.814019 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.814076 (+    57us) service_pool.cc:224] Handling call
0812 08:02:10.814241 (+   165us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:10.814312 (+    71us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:10.814314 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:10.814409 (+    95us) write_op.cc:270] Start()
0812 08:02:10.814433 (+    24us) write_op.cc:276] Timestamp: P: 1786521730812937 usec, L: 0
0812 08:02:10.814805 (+   372us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:10.814845 (+    40us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:10.814851 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:10.814864 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:11.787412 (+972548us) raft_consensus.cc:1685] finished
0812 08:02:11.787415 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:11.787532 (+   117us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:10.814793 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:10.814910 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:10.814914 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:10.814916 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:10.815237 (+   321us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:10.815253 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:10.815256 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:10.815443 (+   187us) tablet.cc:701] Row locks acquired
0812 08:02:10.815444 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":10,"prepare.queue_time_us":256,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":711,"thread_start_us":99,"threads_started":1}]]}
I20260812 08:02:11.788050 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.54]   Dedup: 1.54->[]
W20260812 08:02:11.788105 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 300) took 922 ms (client timeout 50 ms). Trace:
W20260812 08:02:11.788153 31936 rpcz_store.cc:269] 0812 08:02:10.865773 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.866119 (+   346us) service_pool.cc:224] Handling call
0812 08:02:11.787506 (+921387us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:11.787915 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:11.787919 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:11.787932 (+    13us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:11.787936 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.787949 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:11.788064 (+   115us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:11.788092 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2117453696}
W20260812 08:02:11.788389 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 301) took 519 ms (client timeout 50 ms). Trace:
W20260812 08:02:11.788453 31939 rpcz_store.cc:269] 0812 08:02:11.269040 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.269259 (+   219us) service_pool.cc:224] Handling call
0812 08:02:11.788008 (+518749us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:11.788175 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:11.788178 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:11.788187 (+     9us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:11.788190 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.788201 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:11.788339 (+   138us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:11.788373 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1192590208}
I20260812 08:02:11.788753 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.54]   Dedup: 1.54->[]
I20260812 08:02:11.789021 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.54]   Dedup: 1.54->[]
W20260812 08:02:11.789105 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 303) took 85 ms (client timeout 50 ms). Trace:
W20260812 08:02:11.789152 31941 rpcz_store.cc:269] 0812 08:02:11.703596 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.703751 (+   155us) service_pool.cc:224] Handling call
0812 08:02:11.788726 (+ 84975us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:11.788901 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:11.788907 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:11.788917 (+    10us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:11.788924 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.788939 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:11.789065 (+   126us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:11.789094 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194741120}
I20260812 08:02:11.789258 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.54]   Dedup: 1.54->[]
W20260812 08:02:11.789288 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 302) took 469 ms (client timeout 50 ms). Trace:
W20260812 08:02:11.789368 31935 rpcz_store.cc:269] 0812 08:02:11.320254 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.320386 (+   132us) service_pool.cc:224] Handling call
0812 08:02:11.788991 (+468605us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:11.789110 (+   119us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:11.789117 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:11.789125 (+     8us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:11.789130 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.789139 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:11.789247 (+   108us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:11.789267 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1077196544}
I20260812 08:02:11.798265   323 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=52, attempt_no=0}) took 22036 ms. Trace:
I20260812 08:02:11.798442   323 rpcz_store.cc:276] 0812 08:01:49.761965 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:49.762466 (+   501us) service_pool.cc:224] Handling call
0812 08:02:11.798242 (+22035776us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:09.909979 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:09.910111 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:09.910117 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:09.910120 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:09.910453 (+   333us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:09.910463 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:09.910468 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:10.812877 (+902409us) lock_manager.cc:598] Waited 902083us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:10.812893 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:10.812895 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:10.812993 (+    98us) write_op.cc:270] Start()
0812 08:02:10.813021 (+    28us) write_op.cc:276] Timestamp: P: 1786521730812937 usec, L: 0
0812 08:02:10.813023 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:10.813175 (+   152us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:11.795336 (+982161us) op_driver.cc:464] REPLICATION: finished
0812 08:02:11.795657 (+   321us) write_op.cc:301] APPLY: starting
0812 08:02:11.795706 (+    49us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:11.795808 (+   102us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:11.795811 (+     3us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:11.797309 (+  1498us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:11.797373 (+    64us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:11.797379 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:11.797567 (+   188us) log.cc:844] Serialized 319 byte log entry
0812 08:02:11.798046 (+   479us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:11.798135 (+    89us) write_op.cc:454] Released schema lock
0812 08:02:11.798147 (+    12us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":245,"num_ops":50,"prepare.queue_time_us":20146853,"prepare.run_cpu_time_us":1338,"prepare.run_wall_time_us":903384,"raft.queue_time_us":122,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":228,"replication_time_us":982262,"row_lock_wait_count":1,"row_lock_wait_us":902083,"spinlock_wait_cycles":100864,"thread_start_us":125,"threads_started":1}]]}
W20260812 08:02:11.798738 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 952ms of latency in SegmentAllocator::Sync()
W20260812 08:02:11.801563 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1018ms of latency in SegmentAllocator::Sync()
W20260812 08:02:11.901351 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:12.016674 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 289) took 882 ms (client timeout 50 ms). Trace:
I20260812 08:02:12.016728 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.54]   Dedup: 1.54->[]
I20260812 08:02:12.017266 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.54]   Dedup: 1.54->[]
W20260812 08:02:12.016973 32085 rpcz_store.cc:269] 0812 08:02:11.134119 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.134315 (+   196us) service_pool.cc:224] Handling call
0812 08:02:11.134508 (+   193us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:11.134755 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:11.135110 (+   355us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:11.135271 (+   161us) write_op.cc:270] Start()
0812 08:02:11.135299 (+    28us) write_op.cc:276] Timestamp: P: 1786521730812937 usec, L: 0
0812 08:02:11.135644 (+   345us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:11.135691 (+    47us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:11.135697 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.135713 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:12.016536 (+880823us) raft_consensus.cc:1685] finished
0812 08:02:12.016539 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.016641 (+   102us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:11.135622 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:11.135814 (+   192us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:11.135818 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:11.135820 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:11.136156 (+   336us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:11.136167 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:11.136170 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:11.137257 (+  1087us) tablet.cc:701] Row locks acquired
0812 08:02:11.137259 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":215,"prepare.run_cpu_time_us":986,"prepare.run_wall_time_us":1715,"row_lock_wait_count":1,"row_lock_wait_us":790,"thread_start_us":95,"threads_started":1}]]}
W20260812 08:02:12.017522 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 291) took 396 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.017566 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 292) took 345 ms (client timeout 50 ms). Trace:
I20260812 08:02:12.017609 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.54]   Dedup: 1.54->[]
W20260812 08:02:12.017632 32084 rpcz_store.cc:269] 0812 08:02:11.672455 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.672610 (+   155us) service_pool.cc:224] Handling call
0812 08:02:12.017237 (+344627us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:12.017375 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:12.017378 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.017388 (+    10us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:12.017392 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.017403 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.017523 (+   120us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.017552 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":791974272}
W20260812 08:02:12.017593 32086 rpcz_store.cc:269] 0812 08:02:11.621014 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.621120 (+   106us) service_pool.cc:224] Handling call
0812 08:02:12.016682 (+395562us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:12.017130 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:12.017134 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.017147 (+    13us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:12.017153 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.017168 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.017458 (+   290us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.017503 (+    45us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":908973312}
W20260812 08:02:12.017871 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 290) took 832 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.017935 32090 rpcz_store.cc:269] 0812 08:02:11.185537 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.185710 (+   173us) service_pool.cc:224] Handling call
0812 08:02:12.017582 (+831872us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:12.017702 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 53
0812 08:02:12.017707 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.017717 (+    10us) raft_consensus.cc:1645] Marking committed up to 53
0812 08:02:12.017720 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.017730 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.017838 (+   108us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.017860 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1912553728}
I20260812 08:02:12.048709 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.55]   Dedup: 1.54->[1.55-1.55]
W20260812 08:02:12.049998 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 862ms of latency in SegmentAllocator::Sync()
W20260812 08:02:12.724145 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:02:12.799882 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0840
W20260812 08:02:12.820317 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 306) took 1020 ms (client timeout 50 ms). Trace:
I20260812 08:02:12.820317 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.820511 31937 rpcz_store.cc:269] 0812 08:02:11.799956 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.800053 (+    97us) service_pool.cc:224] Handling call
0812 08:02:11.800320 (+   267us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:11.800440 (+   120us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:11.800445 (+     5us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:11.800551 (+   106us) write_op.cc:270] Start()
0812 08:02:11.800578 (+    27us) write_op.cc:276] Timestamp: P: 1786521731798232 usec, L: 0
0812 08:02:11.801115 (+   537us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:11.801359 (+   244us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:11.801369 (+    10us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:11.801387 (+    18us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:12.820090 (+1018703us) raft_consensus.cc:1685] finished
0812 08:02:12.820093 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.820282 (+   189us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:11.801517 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:11.801707 (+   190us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:11.801712 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:11.801715 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:11.802137 (+   422us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:11.802154 (+    17us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:11.802158 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:11.802520 (+   362us) tablet.cc:701] Row locks acquired
0812 08:02:11.802522 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":7,"prepare.queue_time_us":841,"prepare.run_cpu_time_us":1129,"prepare.run_wall_time_us":1129,"thread_start_us":92,"threads_started":1}]]}
I20260812 08:02:12.820940 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.820966 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 307) took 969 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.821064 31935 rpcz_store.cc:269] 0812 08:02:11.851285 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.851393 (+   108us) service_pool.cc:224] Handling call
0812 08:02:12.820265 (+968872us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:12.820569 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.820573 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.820592 (+    19us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.820596 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.820608 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.820730 (+   122us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:12.820932 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227723648}
I20260812 08:02:12.821194 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.821487 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 308) took 444 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.821538 31941 rpcz_store.cc:269] 0812 08:02:12.376605 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.376706 (+   101us) service_pool.cc:224] Handling call
0812 08:02:12.821174 (+444468us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:12.821316 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.821322 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.821332 (+    10us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.821336 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.821345 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.821452 (+   107us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:12.821476 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1021775232}
W20260812 08:02:12.822012 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 309) took 394 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.822069 31939 rpcz_store.cc:269] 0812 08:02:12.427943 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.428104 (+   161us) service_pool.cc:224] Handling call
0812 08:02:12.820896 (+392792us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:12.821098 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.821102 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.821114 (+    12us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.821120 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.821136 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.821968 (+   832us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:12.821999 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902817536}
I20260812 08:02:12.854529 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.55]   Dedup: 1.55->[]
I20260812 08:02:12.859217   335 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=53, attempt_no=0}) took 21896 ms. Trace:
I20260812 08:02:12.859429   335 rpcz_store.cc:276] 0812 08:01:50.962835 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:50.962941 (+   106us) service_pool.cc:224] Handling call
0812 08:02:12.859196 (+21896255us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:10.813311 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:10.813436 (+   125us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:10.813439 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:10.813441 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:10.813725 (+   284us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:10.813734 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:10.813736 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:11.798148 (+984412us) lock_manager.cc:598] Waited 984162us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:11.798164 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:11.798166 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:11.798267 (+   101us) write_op.cc:270] Start()
0812 08:02:11.798295 (+    28us) write_op.cc:276] Timestamp: P: 1786521731798232 usec, L: 0
0812 08:02:11.798298 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:11.798455 (+   157us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:12.856008 (+1057553us) op_driver.cc:464] REPLICATION: finished
0812 08:02:12.856374 (+   366us) write_op.cc:301] APPLY: starting
0812 08:02:12.856422 (+    48us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:12.856501 (+    79us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:12.856503 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:12.858401 (+  1898us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:12.858462 (+    61us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:12.858469 (+     7us) write_op.cc:312] APPLY: finished
0812 08:02:12.858784 (+   315us) log.cc:844] Serialized 319 byte log entry
0812 08:02:12.859034 (+   250us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:12.859094 (+    60us) write_op.cc:454] Released schema lock
0812 08:02:12.859103 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":296,"mutex_wait_us":276,"num_ops":50,"prepare.queue_time_us":19849954,"prepare.run_cpu_time_us":1133,"prepare.run_wall_time_us":985536,"raft.queue_time_us":944,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":211,"replication_time_us":1057669,"row_lock_wait_count":1,"row_lock_wait_us":984162,"spinlock_wait_cycles":107776,"thread_start_us":198,"threads_started":1}]]}
W20260812 08:02:12.859825 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1005ms of latency in SegmentAllocator::Sync()
W20260812 08:02:12.861922 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1145ms of latency in SegmentAllocator::Sync()
W20260812 08:02:12.912606 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 293) took 864 ms (client timeout 50 ms). Trace:
I20260812 08:02:12.912681 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.913040 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 296) took 239 ms (client timeout 50 ms). Trace:
I20260812 08:02:12.913139 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.913098 32085 rpcz_store.cc:269] 0812 08:02:12.673724 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.673817 (+    93us) service_pool.cc:224] Handling call
0812 08:02:12.912630 (+238813us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:12.912845 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.912849 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.912862 (+    13us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.912871 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.912883 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.913001 (+   118us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.913028 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548698496}
W20260812 08:02:12.912735 32090 rpcz_store.cc:269] 0812 08:02:12.048282 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.048427 (+   145us) service_pool.cc:224] Handling call
0812 08:02:12.048645 (+   218us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:12.048860 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.049163 (+   303us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:12.049309 (+   146us) write_op.cc:270] Start()
0812 08:02:12.049337 (+    28us) write_op.cc:276] Timestamp: P: 1786521731798232 usec, L: 0
0812 08:02:12.049709 (+   372us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:12.049754 (+    45us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.049761 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.049777 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:12.912494 (+862717us) raft_consensus.cc:1685] finished
0812 08:02:12.912497 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.912579 (+    82us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:12.049613 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:12.049746 (+   133us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:12.049751 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:12.049752 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:12.049994 (+   242us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:12.050003 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:12.050005 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:12.051732 (+  1727us) tablet.cc:701] Row locks acquired
0812 08:02:12.051735 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":1217,"prepare.run_wall_time_us":2240,"row_lock_wait_count":1,"row_lock_wait_us":1125,"thread_start_us":109,"threads_started":1}]]}
I20260812 08:02:12.913471 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.55]   Dedup: 1.55->[]
W20260812 08:02:12.913542 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 294) took 813 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.913617 32086 rpcz_store.cc:269] 0812 08:02:12.099840 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.099923 (+    83us) service_pool.cc:224] Handling call
0812 08:02:12.912958 (+813035us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:12.913320 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.913326 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.913335 (+     9us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.913339 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.913351 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.913498 (+   147us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.913528 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1869344128}
W20260812 08:02:12.913792 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 295) took 291 ms (client timeout 50 ms). Trace:
W20260812 08:02:12.913897 32084 rpcz_store.cc:269] 0812 08:02:12.622692 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.622925 (+   233us) service_pool.cc:224] Handling call
0812 08:02:12.913440 (+290515us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:12.913582 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 54
0812 08:02:12.913595 (+    13us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:12.913603 (+     8us) raft_consensus.cc:1645] Marking committed up to 54
0812 08:02:12.913608 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.913625 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:12.913744 (+   119us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:12.913779 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":667479424}
W20260812 08:02:12.962106 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:13.200278 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.55->[1.56-1.56]
W20260812 08:02:13.201671 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1134ms of latency in SegmentAllocator::Sync()
W20260812 08:02:13.860447 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb0580
W20260812 08:02:14.007544 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 312) took 1146 ms (client timeout 50 ms). Trace:
I20260812 08:02:14.007579 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.007704 31939 rpcz_store.cc:269] 0812 08:02:12.860721 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.860851 (+   130us) service_pool.cc:224] Handling call
0812 08:02:12.861026 (+   175us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:12.861097 (+    71us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:12.861099 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:12.861202 (+   103us) write_op.cc:270] Start()
0812 08:02:12.861224 (+    22us) write_op.cc:276] Timestamp: P: 1786521732859161 usec, L: 0
0812 08:02:12.861569 (+   345us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:12.861609 (+    40us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:12.861615 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:12.861629 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:14.007406 (+1145777us) raft_consensus.cc:1685] finished
0812 08:02:14.007409 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.007512 (+   103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:12.861517 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:12.861627 (+   110us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:12.861631 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:12.861632 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:12.861991 (+   359us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:12.862002 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:12.862005 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:12.862333 (+   328us) tablet.cc:701] Row locks acquired
0812 08:02:12.862335 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":212,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":883,"thread_start_us":132,"threads_started":1}]]}
I20260812 08:02:14.007947 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.008073 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 315) took 563 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.008273 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 314) took 615 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.008291 31937 rpcz_store.cc:269] 0812 08:02:13.444338 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.444464 (+   126us) service_pool.cc:224] Handling call
0812 08:02:14.007531 (+563067us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.007800 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.007807 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.007822 (+    15us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.007830 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.007846 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.008020 (+   174us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:14.008058 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1294453120}
I20260812 08:02:14.008360 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.008364 31935 rpcz_store.cc:269] 0812 08:02:13.392693 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.392781 (+    88us) service_pool.cc:224] Handling call
0812 08:02:14.007918 (+615137us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.008063 (+   145us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.008066 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.008073 (+     7us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.008077 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.008091 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.008231 (+   140us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:14.008259 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1414291200}
I20260812 08:02:14.008669 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.008749 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 313) took 1096 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.008822 31941 rpcz_store.cc:269] 0812 08:02:12.912057 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.912284 (+   227us) service_pool.cc:224] Handling call
0812 08:02:14.008333 (+1096049us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.008564 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.008567 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.008575 (+     8us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.008579 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.008589 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.008703 (+   114us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:14.008734 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2520220416}
I20260812 08:02:14.008993 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.009057 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 316) took 117 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.009132 31936 rpcz_store.cc:269] 0812 08:02:13.891874 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.892021 (+   147us) service_pool.cc:224] Handling call
0812 08:02:14.008638 (+116617us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.008856 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.008859 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.008868 (+     9us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.008873 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.008888 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.009014 (+   126us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:14.009042 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267740928}
W20260812 08:02:14.009291 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 317) took 66 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.009565 31942 rpcz_store.cc:269] 0812 08:02:13.942872 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.942950 (+    78us) service_pool.cc:224] Handling call
0812 08:02:14.008965 (+ 66015us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.009113 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.009119 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.009129 (+    10us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.009133 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.009143 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.009250 (+   107us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:14.009281 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151369984}
I20260812 08:02:14.284272 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.56]   Dedup: 1.56->[]
I20260812 08:02:14.288972   344 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=54, attempt_no=0}) took 21674 ms. Trace:
W20260812 08:02:14.289322 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1044ms of latency in SegmentAllocator::Sync()
I20260812 08:02:14.289165   344 rpcz_store.cc:276] 0812 08:01:52.614543 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:52.614685 (+   142us) service_pool.cc:224] Handling call
0812 08:02:14.288950 (+21674265us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:11.798872 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:11.798990 (+   118us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:11.798996 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:11.798999 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:11.799329 (+   330us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:11.799336 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:11.799340 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:12.859107 (+1059767us) lock_manager.cc:598] Waited 1059470us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:12.859120 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:12.859121 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:02:12.859203 (+    82us) write_op.cc:270] Start()
0812 08:02:12.859228 (+    25us) write_op.cc:276] Timestamp: P: 1786521732859161 usec, L: 0
0812 08:02:12.859230 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:12.859397 (+   167us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:14.285619 (+1426222us) op_driver.cc:464] REPLICATION: finished
0812 08:02:14.286023 (+   404us) write_op.cc:301] APPLY: starting
0812 08:02:14.286083 (+    60us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:14.286166 (+    83us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:14.286167 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:14.288547 (+  2380us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:14.288597 (+    50us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:14.288601 (+     4us) write_op.cc:312] APPLY: finished
0812 08:02:14.288745 (+   144us) log.cc:844] Serialized 319 byte log entry
0812 08:02:14.288829 (+    84us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:14.288860 (+    31us) write_op.cc:454] Released schema lock
0812 08:02:14.288868 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":337,"num_ops":50,"prepare.queue_time_us":19183687,"prepare.run_cpu_time_us":1390,"prepare.run_wall_time_us":1060617,"raft.queue_time_us":983,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":459,"replication_time_us":1426352,"row_lock_wait_count":1,"row_lock_wait_us":1059470,"spinlock_wait_cycles":76032,"thread_start_us":60,"threads_started":1}]]}
W20260812 08:02:14.291137 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1084ms of latency in SegmentAllocator::Sync()
W20260812 08:02:14.336477 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 297) took 1136 ms (client timeout 50 ms). Trace:
I20260812 08:02:14.336524 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.336817 32084 rpcz_store.cc:269] 0812 08:02:13.199811 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.199907 (+    96us) service_pool.cc:224] Handling call
0812 08:02:13.200206 (+   299us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:13.200434 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:13.200685 (+   251us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:13.200830 (+   145us) write_op.cc:270] Start()
0812 08:02:13.200859 (+    29us) write_op.cc:276] Timestamp: P: 1786521732859161 usec, L: 0
0812 08:02:13.201198 (+   339us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:13.201243 (+    45us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:13.201249 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:13.201263 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:14.336333 (+1135070us) raft_consensus.cc:1685] finished
0812 08:02:14.336336 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.336446 (+   110us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:13.201102 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:13.201228 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:13.201233 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:13.201234 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:13.201557 (+   323us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:13.201583 (+    26us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:13.201586 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:13.203100 (+  1514us) tablet.cc:701] Row locks acquired
0812 08:02:13.203102 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":986,"prepare.run_wall_time_us":2069,"row_lock_wait_count":1,"row_lock_wait_us":1160,"thread_start_us":88,"threads_started":1}]]}
I20260812 08:02:14.337268 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.337303 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 298) took 1086 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.337378 32086 rpcz_store.cc:269] 0812 08:02:13.251041 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.251212 (+   171us) service_pool.cc:224] Handling call
0812 08:02:14.336483 (+1085271us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:14.337082 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.337089 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.337101 (+    12us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.337105 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.337141 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.337255 (+   114us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:14.337286 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2495261056}
I20260812 08:02:14.337498 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.337550 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 299) took 752 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.337656 32085 rpcz_store.cc:269] 0812 08:02:13.584780 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.584899 (+   119us) service_pool.cc:224] Handling call
0812 08:02:14.337244 (+752345us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:14.337389 (+   145us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.337392 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.337399 (+     7us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.337403 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.337412 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.337517 (+   105us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:14.337540 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1729673472}
I20260812 08:02:14.337783 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.337862 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 300) took 702 ms (client timeout 50 ms). Trace:
I20260812 08:02:14.338014 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.56]   Dedup: 1.56->[]
W20260812 08:02:14.338027 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 302) took 107 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.337975 32090 rpcz_store.cc:269] 0812 08:02:13.635847 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:13.636078 (+   231us) service_pool.cc:224] Handling call
0812 08:02:14.337462 (+701384us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:14.337669 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.337674 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.337685 (+    11us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.337691 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.337706 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.337823 (+   117us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:14.337850 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612522240}
W20260812 08:02:14.338119 32087 rpcz_store.cc:269] 0812 08:02:14.230498 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.230822 (+   324us) service_pool.cc:224] Handling call
0812 08:02:14.337751 (+106929us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:14.337866 (+   115us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.337872 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.337880 (+     8us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.337884 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.337893 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.337997 (+   104us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:14.338016 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245401856}
W20260812 08:02:14.338330 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 301) took 158 ms (client timeout 50 ms). Trace:
W20260812 08:02:14.338518 32089 rpcz_store.cc:269] 0812 08:02:14.179413 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.179603 (+   190us) service_pool.cc:224] Handling call
0812 08:02:14.337983 (+158380us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:14.338144 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 55
0812 08:02:14.338149 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:14.338158 (+     9us) raft_consensus.cc:1645] Marking committed up to 55
0812 08:02:14.338162 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.338171 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:14.338287 (+   116us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:14.338319 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363780480}
W20260812 08:02:14.390830 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:14.793131 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.57]   Dedup: 1.56->[1.57-1.57]
W20260812 08:02:14.794855 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 854ms of latency in SegmentAllocator::Sync()
W20260812 08:02:15.172776 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 22 similar messages]
W20260812 08:02:15.290269 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb02c0
W20260812 08:02:15.375933 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 320) took 1086 ms (client timeout 50 ms). Trace:
I20260812 08:02:15.375936 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.376111 31942 rpcz_store.cc:269] 0812 08:02:14.289853 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.289935 (+    82us) service_pool.cc:224] Handling call
0812 08:02:14.290093 (+   158us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:14.290158 (+    65us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:14.290160 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:14.290309 (+   149us) write_op.cc:270] Start()
0812 08:02:14.290336 (+    27us) write_op.cc:276] Timestamp: P: 1786521734288862 usec, L: 0
0812 08:02:14.290692 (+   356us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:14.290743 (+    51us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:14.290753 (+    10us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.290766 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:15.375703 (+1084937us) raft_consensus.cc:1685] finished
0812 08:02:15.375705 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.375889 (+   184us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:14.290620 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:14.290726 (+   106us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:14.290730 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:14.290731 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:14.290990 (+   259us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:14.290998 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:14.291000 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:14.291169 (+   169us) tablet.cc:701] Row locks acquired
0812 08:02:14.291170 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":673,"prepare.run_wall_time_us":673,"thread_start_us":94,"threads_started":1}]]}
I20260812 08:02:15.376349 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.376372 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 323) took 639 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.376502 31937 rpcz_store.cc:269] 0812 08:02:14.737270 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.737363 (+    93us) service_pool.cc:224] Handling call
0812 08:02:15.375864 (+638501us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.376160 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.376163 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.376175 (+    12us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.376179 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.376190 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.376333 (+   143us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:15.376359 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1467959552}
I20260812 08:02:15.376629 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.376688 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 324) took 254 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.376735 31935 rpcz_store.cc:269] 0812 08:02:15.122479 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.122770 (+   291us) service_pool.cc:224] Handling call
0812 08:02:15.376306 (+253536us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.376515 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.376518 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.376527 (+     9us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.376531 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.376541 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.376653 (+   112us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:15.376677 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582246784}
I20260812 08:02:15.376889 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.376894 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 325) took 202 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.377053 31939 rpcz_store.cc:269] 0812 08:02:15.174039 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.174217 (+   178us) service_pool.cc:224] Handling call
0812 08:02:15.376602 (+202385us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.376724 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.376726 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.376734 (+     8us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.376739 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.376749 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.376857 (+   108us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:15.376883 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":465014016}
W20260812 08:02:15.377178 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 321) took 1036 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.377252 31936 rpcz_store.cc:269] 0812 08:02:14.340669 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.340824 (+   155us) service_pool.cc:224] Handling call
0812 08:02:15.376865 (+1036041us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.376990 (+   125us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.376994 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.377001 (+     7us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.377008 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.377022 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.377134 (+   112us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:15.377167 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2381809408}
I20260812 08:02:15.377262 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.377619 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 322) took 692 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.377686 31941 rpcz_store.cc:269] 0812 08:02:14.685484 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.685596 (+   112us) service_pool.cc:224] Handling call
0812 08:02:15.377232 (+691636us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.377457 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.377461 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.377470 (+     9us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.377476 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.377490 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.377588 (+    98us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:15.377609 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589872512}
W20260812 08:02:15.494784 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 11: this message will repeat every 5th retry.
I20260812 08:02:15.637176 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57]   Dedup: 1.57->[]
I20260812 08:02:15.641463   352 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=55, attempt_no=0}) took 21898 ms. Trace:
W20260812 08:02:15.641845 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1125ms of latency in SegmentAllocator::Sync()
I20260812 08:02:15.641652   352 rpcz_store.cc:276] 0812 08:01:53.743073 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:53.743190 (+   117us) service_pool.cc:224] Handling call
0812 08:02:15.641244 (+21898054us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:12.859518 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:12.859645 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:12.859650 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:12.859652 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:12.859997 (+   345us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:12.860005 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:12.860009 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:14.288899 (+1428890us) lock_manager.cc:598] Waited 1428580us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:14.288913 (+    14us) tablet.cc:701] Row locks acquired
0812 08:02:14.288915 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:14.288975 (+    60us) write_op.cc:270] Start()
0812 08:02:14.288999 (+    24us) write_op.cc:276] Timestamp: P: 1786521734288862 usec, L: 0
0812 08:02:14.289002 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:14.289141 (+   139us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:15.638415 (+1349274us) op_driver.cc:464] REPLICATION: finished
0812 08:02:15.638885 (+   470us) write_op.cc:301] APPLY: starting
0812 08:02:15.638955 (+    70us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:15.639067 (+   112us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:15.639069 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:15.640856 (+  1787us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:15.640915 (+    59us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:15.640921 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:15.641048 (+   127us) log.cc:844] Serialized 319 byte log entry
0812 08:02:15.641117 (+    69us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:15.641146 (+    29us) write_op.cc:454] Released schema lock
0812 08:02:15.641154 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":223,"num_ops":50,"prepare.queue_time_us":19115991,"prepare.run_cpu_time_us":1395,"prepare.run_wall_time_us":1429735,"raft.queue_time_us":113,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":172,"replication_time_us":1349375,"row_lock_wait_count":1,"row_lock_wait_us":1428580,"spinlock_wait_cycles":10496,"thread_start_us":93,"threads_started":1}]]}
W20260812 08:02:15.643600 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1079ms of latency in SegmentAllocator::Sync()
W20260812 08:02:15.649488 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 303) took 856 ms (client timeout 50 ms). Trace:
I20260812 08:02:15.649520 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.57]   Dedup: 1.57->[]
I20260812 08:02:15.649884 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.649994 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 305) took 256 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.650045 32090 rpcz_store.cc:269] 0812 08:02:15.393552 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.393687 (+   135us) service_pool.cc:224] Handling call
0812 08:02:15.649474 (+255787us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:15.649757 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.649760 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.649771 (+    11us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.649775 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.649791 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.649944 (+   153us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:15.649980 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":587725056}
W20260812 08:02:15.650151 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 304) took 806 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.649610 32089 rpcz_store.cc:269] 0812 08:02:14.792709 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.792848 (+   139us) service_pool.cc:224] Handling call
0812 08:02:14.793071 (+   223us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:14.793332 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:14.793559 (+   227us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:14.794024 (+   465us) write_op.cc:270] Start()
0812 08:02:14.794057 (+    33us) write_op.cc:276] Timestamp: P: 1786521734288862 usec, L: 0
0812 08:02:14.794454 (+   397us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:14.794503 (+    49us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:14.794510 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:14.794525 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:15.649361 (+854836us) raft_consensus.cc:1685] finished
0812 08:02:15.649363 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.649457 (+    94us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:14.794351 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:14.794481 (+   130us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:14.794487 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:14.794489 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:14.795024 (+   535us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:14.795042 (+    18us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:14.795048 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:14.795562 (+   514us) tablet.cc:701] Row locks acquired
0812 08:02:14.795564 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":203,"prepare.run_cpu_time_us":1083,"prepare.run_wall_time_us":1276,"row_lock_wait_count":1,"row_lock_wait_us":272,"thread_start_us":133,"threads_started":1}]]}
W20260812 08:02:15.650188 32087 rpcz_store.cc:269] 0812 08:02:14.843887 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:14.844017 (+   130us) service_pool.cc:224] Handling call
0812 08:02:15.649852 (+805835us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:15.649999 (+   147us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.650004 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.650013 (+     9us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.650017 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.650025 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.650123 (+    98us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:15.650141 (+    18us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1852701056}
I20260812 08:02:15.650436 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.57]   Dedup: 1.57->[]
W20260812 08:02:15.651381 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 306) took 206 ms (client timeout 50 ms). Trace:
W20260812 08:02:15.651654 32085 rpcz_store.cc:269] 0812 08:02:15.444362 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.444484 (+   122us) service_pool.cc:224] Handling call
0812 08:02:15.650405 (+205921us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:15.650972 (+   567us) raft_consensus.cc:1534] Early marking committed up to index 56
0812 08:02:15.650989 (+    17us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:15.651022 (+    33us) raft_consensus.cc:1645] Marking committed up to 56
0812 08:02:15.651033 (+    11us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.651060 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:15.651220 (+   160us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:15.651279 (+    59us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":472846208}
W20260812 08:02:15.744102 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:15.963979 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.57->[1.58-1.58]
W20260812 08:02:15.965480 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1040ms of latency in SegmentAllocator::Sync()
W20260812 08:02:16.642859 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb18c0
I20260812 08:02:16.723412 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
W20260812 08:02:16.723412 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 328) took 1080 ms (client timeout 50 ms). Trace:
I20260812 08:02:16.723902 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
W20260812 08:02:16.724025 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 331) took 612 ms (client timeout 50 ms). Trace:
W20260812 08:02:16.723822 31941 rpcz_store.cc:269] 0812 08:02:15.642542 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.642683 (+   141us) service_pool.cc:224] Handling call
0812 08:02:15.642853 (+   170us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:15.642931 (+    78us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:15.642934 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:15.643050 (+   116us) write_op.cc:270] Start()
0812 08:02:15.643072 (+    22us) write_op.cc:276] Timestamp: P: 1786521735641181 usec, L: 0
0812 08:02:15.643386 (+   314us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:15.643423 (+    37us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:15.643428 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.643440 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:16.723141 (+1079701us) raft_consensus.cc:1685] finished
0812 08:02:16.723145 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.723366 (+   221us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:15.643331 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:15.643440 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:15.643443 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:15.643445 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:15.643736 (+   291us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:15.643745 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:15.643747 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:15.643909 (+   162us) tablet.cc:701] Row locks acquired
0812 08:02:15.643910 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":637,"thread_start_us":102,"threads_started":1}]]}
I20260812 08:02:16.724251 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
W20260812 08:02:16.724253 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 330) took 663 ms (client timeout 50 ms). Trace:
I20260812 08:02:16.724788 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
W20260812 08:02:16.724625 31939 rpcz_store.cc:269] 0812 08:02:16.060334 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.060418 (+    84us) service_pool.cc:224] Handling call
0812 08:02:16.723871 (+663453us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.724053 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:16.724056 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:16.724066 (+    10us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:16.724072 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.724087 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.724214 (+   127us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:16.724240 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525394816}
W20260812 08:02:16.724720 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 329) took 1030 ms (client timeout 50 ms). Trace:
I20260812 08:02:16.725029 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
W20260812 08:02:16.724974 31936 rpcz_store.cc:269] 0812 08:02:15.694067 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.694224 (+   157us) service_pool.cc:224] Handling call
0812 08:02:16.724185 (+1029961us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.724441 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:16.724445 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:16.724459 (+    14us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:16.724463 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.724482 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.724658 (+   176us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:16.724703 (+    45us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2368173952}
W20260812 08:02:16.725111 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 333) took 194 ms (client timeout 50 ms). Trace:
W20260812 08:02:16.725237 31942 rpcz_store.cc:269] 0812 08:02:16.530923 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.531005 (+    82us) service_pool.cc:224] Handling call
0812 08:02:16.724753 (+193748us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.724923 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:16.724925 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:16.724933 (+     8us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:16.724936 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.724947 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.725070 (+   123us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:16.725098 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445088128}
W20260812 08:02:16.725324 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 332) took 245 ms (client timeout 50 ms). Trace:
W20260812 08:02:16.724283 31935 rpcz_store.cc:269] 0812 08:02:16.111547 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.111694 (+   147us) service_pool.cc:224] Handling call
0812 08:02:16.723334 (+611640us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.723665 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:16.723667 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:16.723681 (+    14us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:16.723684 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.723696 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.723970 (+   274us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:16.724009 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1406066048}
W20260812 08:02:16.725427 31937 rpcz_store.cc:269] 0812 08:02:16.479925 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.480077 (+   152us) service_pool.cc:224] Handling call
0812 08:02:16.725000 (+244923us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.725121 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:16.725125 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:16.725134 (+     9us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:16.725138 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.725149 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:16.725273 (+   124us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:16.725304 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562828544}
I20260812 08:02:16.974475 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.58]   Dedup: 1.58->[]
I20260812 08:02:16.978227   360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=56, attempt_no=0}) took 22191 ms. Trace:
I20260812 08:02:16.978433   360 rpcz_store.cc:276] 0812 08:01:54.787208 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:54.787316 (+   108us) service_pool.cc:224] Handling call
0812 08:02:16.978206 (+22190890us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:14.289284 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:14.289401 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:14.289405 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:14.289406 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:14.289835 (+   429us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:14.289856 (+    21us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:14.289859 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:15.641208 (+1351349us) lock_manager.cc:598] Waited 1351063us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:15.641225 (+    17us) tablet.cc:701] Row locks acquired
0812 08:02:15.641229 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:02:15.641281 (+    52us) write_op.cc:270] Start()
0812 08:02:15.641311 (+    30us) write_op.cc:276] Timestamp: P: 1786521735641181 usec, L: 0
0812 08:02:15.641313 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:15.641484 (+   171us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:16.975958 (+1334474us) op_driver.cc:464] REPLICATION: finished
0812 08:02:16.976264 (+   306us) write_op.cc:301] APPLY: starting
0812 08:02:16.976302 (+    38us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:16.976365 (+    63us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:16.976366 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:16.977781 (+  1415us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:16.977838 (+    57us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:16.977844 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:16.977974 (+   130us) log.cc:844] Serialized 319 byte log entry
0812 08:02:16.978044 (+    70us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:16.978073 (+    29us) write_op.cc:454] Released schema lock
0812 08:02:16.978082 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":240,"num_ops":50,"prepare.queue_time_us":19501489,"prepare.run_cpu_time_us":1408,"prepare.run_wall_time_us":1352287,"raft.queue_time_us":945,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":235,"replication_time_us":1334600,"row_lock_wait_count":1,"row_lock_wait_us":1351063,"thread_start_us":99,"threads_started":1}]]}
W20260812 08:02:16.979185 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1056ms of latency in SegmentAllocator::Sync()
W20260812 08:02:16.981038 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 810ms of latency in SegmentAllocator::Sync()
W20260812 08:02:17.006105 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 307) took 1042 ms (client timeout 50 ms). Trace:
I20260812 08:02:17.006263 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.58->[]
W20260812 08:02:17.006234 32085 rpcz_store.cc:269] 0812 08:02:15.963516 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:15.963675 (+   159us) service_pool.cc:224] Handling call
0812 08:02:15.963914 (+   239us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:15.964431 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:15.964715 (+   284us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:15.964842 (+   127us) write_op.cc:270] Start()
0812 08:02:15.964870 (+    28us) write_op.cc:276] Timestamp: P: 1786521735641181 usec, L: 0
0812 08:02:15.965190 (+   320us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:15.965234 (+    44us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:15.965239 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:15.965270 (+    31us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:17.005976 (+1040706us) raft_consensus.cc:1685] finished
0812 08:02:17.005979 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.006074 (+    95us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:15.965107 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:15.965233 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:15.965237 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:15.965239 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:15.965483 (+   244us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:15.965492 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:15.965495 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:15.966804 (+  1309us) tablet.cc:701] Row locks acquired
0812 08:02:15.966806 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":1774,"row_lock_wait_count":1,"row_lock_wait_us":1003,"thread_start_us":83,"threads_started":1}]]}
I20260812 08:02:17.006759 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.58->[]
W20260812 08:02:17.006956 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 310) took 577 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.007031 32087 rpcz_store.cc:269] 0812 08:02:16.429647 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.429862 (+   215us) service_pool.cc:224] Handling call
0812 08:02:17.006205 (+576343us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:17.006550 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:17.006553 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.006587 (+    34us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:17.006620 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.006646 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.006866 (+   220us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:17.006937 (+    71us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1324909312}
I20260812 08:02:17.007128 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.58->[]
W20260812 08:02:17.007136 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 309) took 628 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.007426 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 308) took 992 ms (client timeout 50 ms). Trace:
I20260812 08:02:17.007579 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.58->[]
W20260812 08:02:17.007535 32090 rpcz_store.cc:269] 0812 08:02:16.015143 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.015256 (+   113us) service_pool.cc:224] Handling call
0812 08:02:17.007038 (+991782us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:17.007233 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:17.007236 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.007244 (+     8us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:17.007250 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.007260 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.007370 (+   110us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:17.007392 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2280279424}
I20260812 08:02:17.007854 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.58]   Dedup: 1.58->[]
W20260812 08:02:17.007386 32089 rpcz_store.cc:269] 0812 08:02:16.378528 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.378625 (+    97us) service_pool.cc:224] Handling call
0812 08:02:17.006730 (+628105us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:17.006956 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:17.006959 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.006966 (+     7us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:17.006971 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.006982 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.007098 (+   116us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:17.007123 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1443580288}
W20260812 08:02:17.008256 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 311) took 182 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.008312 32086 rpcz_store.cc:269] 0812 08:02:16.826075 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.826155 (+    80us) service_pool.cc:224] Handling call
0812 08:02:17.007832 (+181677us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:17.007969 (+   137us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:17.007972 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.008015 (+    43us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:17.008020 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.008048 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.008198 (+   150us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:17.008232 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417291136}
W20260812 08:02:17.008518 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 312) took 130 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.008581 32084 rpcz_store.cc:269] 0812 08:02:16.877784 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.877908 (+   124us) service_pool.cc:224] Handling call
0812 08:02:17.007515 (+129607us) raft_consensus.cc:1471] Updating replica for 6 ops
0812 08:02:17.007751 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 57
0812 08:02:17.007754 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.007761 (+     7us) raft_consensus.cc:1645] Marking committed up to 57
0812 08:02:17.007765 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.007775 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.008477 (+   702us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:17.008507 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297458432}
W20260812 08:02:17.081771 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:17.316676 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.59]   Dedup: 1.58->[1.59-1.59]
W20260812 08:02:17.318188 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 752ms of latency in SegmentAllocator::Sync()
W20260812 08:02:17.791810 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 336) took 812 ms (client timeout 50 ms). Trace:
I20260812 08:02:17.791829 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.59]   Dedup: 1.59->[]
I20260812 08:02:17.792601 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.59]   Dedup: 1.59->[]
W20260812 08:02:17.792243 31937 rpcz_store.cc:269] 0812 08:02:16.979844 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:16.980010 (+   166us) service_pool.cc:224] Handling call
0812 08:02:16.980185 (+   175us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:16.980261 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:16.980263 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:16.980375 (+   112us) write_op.cc:270] Start()
0812 08:02:16.980398 (+    23us) write_op.cc:276] Timestamp: P: 1786521736978376 usec, L: 0
0812 08:02:16.980730 (+   332us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:16.980850 (+   120us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:16.980858 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:16.980873 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:17.791532 (+810659us) raft_consensus.cc:1685] finished
0812 08:02:17.791535 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.791775 (+   240us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:16.980657 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:16.980852 (+   195us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:16.980856 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:16.980858 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:16.981130 (+   272us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:16.981140 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:16.981142 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:16.981386 (+   244us) tablet.cc:701] Row locks acquired
0812 08:02:16.981388 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":190,"prepare.run_cpu_time_us":793,"prepare.run_wall_time_us":793,"thread_start_us":106,"threads_started":1}]]}
W20260812 08:02:17.792980 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 337) took 761 ms (client timeout 50 ms). Trace:
I20260812 08:02:17.793061 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.59]   Dedup: 1.59->[]
W20260812 08:02:17.793203 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 338) took 415 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.793330 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 339) took 364 ms (client timeout 50 ms). Trace:
W20260812 08:02:17.793326 31942 rpcz_store.cc:269] 0812 08:02:17.377538 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.377792 (+   254us) service_pool.cc:224] Handling call
0812 08:02:17.792556 (+414764us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:17.792825 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:17.792829 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.792871 (+    42us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:17.792877 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.792893 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.793160 (+   267us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:17.793190 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":953411328}
W20260812 08:02:17.793433 31936 rpcz_store.cc:269] 0812 08:02:17.429056 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.429157 (+   101us) service_pool.cc:224] Handling call
0812 08:02:17.793033 (+363876us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:17.793182 (+   149us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:17.793185 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.793192 (+     7us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:17.793195 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.793204 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.793301 (+    97us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:17.793322 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":836261248}
W20260812 08:02:17.793066 31935 rpcz_store.cc:269] 0812 08:02:17.031670 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.031810 (+   140us) service_pool.cc:224] Handling call
0812 08:02:17.791772 (+759962us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:17.792331 (+   559us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:17.792336 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:17.792356 (+    20us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:17.792360 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.792379 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:17.792914 (+   535us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:17.792963 (+    49us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1747273088}
I20260812 08:02:17.831197 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.59]   Dedup: 1.59->[]
W20260812 08:02:17.873356 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 16: this message will repeat every 5th retry.
W20260812 08:02:17.979754 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8b00
I20260812 08:02:18.038374   367 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=57, attempt_no=0}) took 22060 ms. Trace:
I20260812 08:02:18.038571   367 rpcz_store.cc:276] 0812 08:01:55.977688 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:55.977903 (+   215us) service_pool.cc:224] Handling call
0812 08:02:18.038353 (+22060450us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:15.641598 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:15.641905 (+   307us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:15.641911 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:15.641919 (+     8us) tablet.cc:662] Decoding operations
0812 08:02:15.642236 (+   317us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:15.642245 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:15.642249 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:16.978370 (+1336121us) lock_manager.cc:598] Waited 1335660us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:16.978384 (+    14us) tablet.cc:701] Row locks acquired
0812 08:02:16.978386 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:16.978431 (+    45us) write_op.cc:270] Start()
0812 08:02:16.978461 (+    30us) write_op.cc:276] Timestamp: P: 1786521736978376 usec, L: 0
0812 08:02:16.978463 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:16.978652 (+   189us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:18.035888 (+1057236us) op_driver.cc:464] REPLICATION: finished
0812 08:02:18.036296 (+   408us) write_op.cc:301] APPLY: starting
0812 08:02:18.036332 (+    36us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:18.036386 (+    54us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:18.036387 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:18.037758 (+  1371us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:18.037832 (+    74us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:18.037838 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:18.037961 (+   123us) log.cc:844] Serialized 319 byte log entry
0812 08:02:18.038224 (+   263us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:18.038261 (+    37us) write_op.cc:454] Released schema lock
0812 08:02:18.038271 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":338,"num_ops":50,"prepare.queue_time_us":19663277,"prepare.run_cpu_time_us":1768,"prepare.run_wall_time_us":1337144,"raft.queue_time_us":744,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":236,"replication_time_us":1057371,"row_lock_wait_count":1,"row_lock_wait_us":1335660,"thread_start_us":137,"threads_started":1}]]}
W20260812 08:02:18.039150 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 799ms of latency in SegmentAllocator::Sync()
W20260812 08:02:18.040449 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1088ms of latency in SegmentAllocator::Sync()
W20260812 08:02:18.070925 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 313) took 754 ms (client timeout 50 ms). Trace:
I20260812 08:02:18.071034 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.59]   Dedup: 1.59->[]
W20260812 08:02:18.071395 32084 rpcz_store.cc:269] 0812 08:02:17.316208 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.316380 (+   172us) service_pool.cc:224] Handling call
0812 08:02:17.316603 (+   223us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:17.316844 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:17.317302 (+   458us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:17.317439 (+   137us) write_op.cc:270] Start()
0812 08:02:17.317467 (+    28us) write_op.cc:276] Timestamp: P: 1786521736978376 usec, L: 0
0812 08:02:17.317880 (+   413us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:17.317944 (+    64us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:17.317950 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:17.317964 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:18.070775 (+752811us) raft_consensus.cc:1685] finished
0812 08:02:18.070778 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:18.070892 (+   114us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:17.317765 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:17.317917 (+   152us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:17.317923 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:17.317926 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:17.318239 (+   313us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:17.318250 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:17.318253 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:17.319406 (+  1153us) tablet.cc:701] Row locks acquired
0812 08:02:17.319408 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":217,"prepare.run_cpu_time_us":1047,"prepare.run_wall_time_us":1709,"row_lock_wait_count":1,"row_lock_wait_us":812,"thread_start_us":157,"threads_started":1}]]}
I20260812 08:02:18.071704 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.59]   Dedup: 1.59->[]
W20260812 08:02:18.071782 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 314) took 704 ms (client timeout 50 ms). Trace:
I20260812 08:02:18.071942 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.59]   Dedup: 1.59->[]
W20260812 08:02:18.071990 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 316) took 248 ms (client timeout 50 ms). Trace:
W20260812 08:02:18.071866 32086 rpcz_store.cc:269] 0812 08:02:17.367474 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.367600 (+   126us) service_pool.cc:224] Handling call
0812 08:02:18.070972 (+703372us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:18.071535 (+   563us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:18.071542 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:18.071557 (+    15us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:18.071562 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:18.071590 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:18.071723 (+   133us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:18.071762 (+    39us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1616825216}
W20260812 08:02:18.072038 32090 rpcz_store.cc:269] 0812 08:02:17.823795 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.823966 (+   171us) service_pool.cc:224] Handling call
0812 08:02:18.071671 (+247705us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:18.071797 (+   126us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:18.071803 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:18.071812 (+     9us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:18.071816 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:18.071825 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:18.071947 (+   122us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:18.071977 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568297344}
W20260812 08:02:18.072274 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 315) took 300 ms (client timeout 50 ms). Trace:
W20260812 08:02:18.072350 32089 rpcz_store.cc:269] 0812 08:02:17.772132 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:17.772312 (+   180us) service_pool.cc:224] Handling call
0812 08:02:18.071915 (+299603us) raft_consensus.cc:1471] Updating replica for 7 ops
0812 08:02:18.072057 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 58
0812 08:02:18.072063 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:18.072074 (+    11us) raft_consensus.cc:1645] Marking committed up to 58
0812 08:02:18.072080 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:18.072091 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:18.072223 (+   132us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:18.072259 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":688462720}
W20260812 08:02:18.140648 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:18.388548 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.60]   Dedup: 1.59->[1.60-1.60]
W20260812 08:02:18.389719 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1049ms of latency in SegmentAllocator::Sync()
W20260812 08:02:19.039462 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8dc0
W20260812 08:02:19.129050 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 342) took 1089 ms (client timeout 50 ms). Trace:
I20260812 08:02:19.129112 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.129168 31935 rpcz_store.cc:269] 0812 08:02:18.039518 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.039578 (+    60us) service_pool.cc:224] Handling call
0812 08:02:18.039741 (+   163us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:18.039839 (+    98us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:18.039843 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:18.039964 (+   121us) write_op.cc:270] Start()
0812 08:02:18.039984 (+    20us) write_op.cc:276] Timestamp: P: 1786521738038321 usec, L: 0
0812 08:02:18.040235 (+   251us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:18.040296 (+    61us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:18.040302 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:18.040314 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:19.128917 (+1088603us) raft_consensus.cc:1685] finished
0812 08:02:19.128921 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.129022 (+   101us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:18.040312 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:18.040410 (+    98us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:18.040413 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:18.040414 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:18.040884 (+   470us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:18.040895 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:18.040898 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:18.041084 (+   186us) tablet.cc:701] Row locks acquired
0812 08:02:18.041085 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":252,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":841,"thread_start_us":77,"threads_started":1}]]}
I20260812 08:02:19.129815 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.129916 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 345) took 476 ms (client timeout 50 ms). Trace:
I20260812 08:02:19.130064 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.130110 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 344) took 528 ms (client timeout 50 ms). Trace:
W20260812 08:02:19.129969 31937 rpcz_store.cc:269] 0812 08:02:18.653869 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.654011 (+   142us) service_pool.cc:224] Handling call
0812 08:02:19.129057 (+475046us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:19.129546 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.129549 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.129560 (+    11us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.129564 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.129575 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.129864 (+   289us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:19.129900 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1091881856}
W20260812 08:02:19.130209 31942 rpcz_store.cc:269] 0812 08:02:18.602056 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.602146 (+    90us) service_pool.cc:224] Handling call
0812 08:02:19.129774 (+527628us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:19.129942 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.129946 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.129953 (+     7us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.129958 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.129970 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.130074 (+   104us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:19.130098 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1212959488}
W20260812 08:02:19.130476 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 343) took 1039 ms (client timeout 50 ms). Trace:
W20260812 08:02:19.130565 31936 rpcz_store.cc:269] 0812 08:02:18.090885 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.091038 (+   153us) service_pool.cc:224] Handling call
0812 08:02:19.130031 (+1038993us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:19.130243 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.130249 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.130261 (+    12us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.130269 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.130284 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.130426 (+   142us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:19.130460 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2388924800}
I20260812 08:02:19.210083 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.60]   Dedup: 1.60->[]
I20260812 08:02:19.213433   378 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=58, attempt_no=0}) took 21874 ms. Trace:
I20260812 08:02:19.213596   378 rpcz_store.cc:276] 0812 08:01:57.339189 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:57.339307 (+   118us) service_pool.cc:224] Handling call
0812 08:02:19.213410 (+21874103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:16.978779 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:16.978902 (+   123us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:16.978908 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:16.978909 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:16.979242 (+   333us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:16.979251 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:16.979254 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:18.038304 (+1059050us) lock_manager.cc:598] Waited 1058787us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:18.038320 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:18.038322 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:18.038375 (+    53us) write_op.cc:270] Start()
0812 08:02:18.038401 (+    26us) write_op.cc:276] Timestamp: P: 1786521738038321 usec, L: 0
0812 08:02:18.038403 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:18.038551 (+   148us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:19.211404 (+1172853us) op_driver.cc:464] REPLICATION: finished
0812 08:02:19.211780 (+   376us) write_op.cc:301] APPLY: starting
0812 08:02:19.211815 (+    35us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:19.211887 (+    72us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:19.211888 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:19.213069 (+  1181us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:19.213110 (+    41us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:19.213114 (+     4us) write_op.cc:312] APPLY: finished
0812 08:02:19.213210 (+    96us) log.cc:844] Serialized 319 byte log entry
0812 08:02:19.213292 (+    82us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:19.213322 (+    30us) write_op.cc:454] Released schema lock
0812 08:02:19.213330 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":313,"num_ops":50,"prepare.queue_time_us":19639080,"prepare.run_cpu_time_us":1366,"prepare.run_wall_time_us":1059879,"raft.queue_time_us":237,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":165,"replication_time_us":1172927,"row_lock_wait_count":1,"row_lock_wait_us":1058787,"spinlock_wait_cycles":896,"thread_start_us":90,"threads_started":1}]]}
W20260812 08:02:19.214675 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 922ms of latency in SegmentAllocator::Sync()
W20260812 08:02:19.215476 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1031ms of latency in SegmentAllocator::Sync()
W20260812 08:02:19.315899 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:19.439319 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 317) took 1051 ms (client timeout 50 ms). Trace:
I20260812 08:02:19.439319 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.439498 32089 rpcz_store.cc:269] 0812 08:02:18.388096 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.388224 (+   128us) service_pool.cc:224] Handling call
0812 08:02:18.388485 (+   261us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:18.388758 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:18.388973 (+   215us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:18.389098 (+   125us) write_op.cc:270] Start()
0812 08:02:18.389125 (+    27us) write_op.cc:276] Timestamp: P: 1786521738038321 usec, L: 0
0812 08:02:18.389439 (+   314us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:18.389481 (+    42us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:18.389486 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:18.389498 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:19.439173 (+1049675us) raft_consensus.cc:1685] finished
0812 08:02:19.439178 (+     5us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.439293 (+   115us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:18.389352 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:18.389474 (+   122us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:18.389478 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:18.389480 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:18.389696 (+   216us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:18.389705 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:18.389707 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:18.390891 (+  1184us) tablet.cc:701] Row locks acquired
0812 08:02:18.390894 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":1604,"row_lock_wait_count":1,"row_lock_wait_us":887,"thread_start_us":92,"threads_started":1}]]}
W20260812 08:02:19.439723 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 318) took 1000 ms (client timeout 50 ms). Trace:
W20260812 08:02:19.439775 32090 rpcz_store.cc:269] 0812 08:02:18.439101 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.439199 (+    98us) service_pool.cc:224] Handling call
0812 08:02:19.439266 (+1000067us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:19.439559 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.439562 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.439574 (+    12us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.439577 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.439588 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.439689 (+   101us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:19.439712 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2299196800}
I20260812 08:02:19.439677 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.440095 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 319) took 600 ms (client timeout 50 ms). Trace:
W20260812 08:02:19.440165 32086 rpcz_store.cc:269] 0812 08:02:18.839808 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.839962 (+   154us) service_pool.cc:224] Handling call
0812 08:02:19.439640 (+599678us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:19.439923 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.439927 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.439934 (+     7us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.439937 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.439948 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.440055 (+   107us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:19.440083 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1378569472}
I20260812 08:02:19.440224 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.60]   Dedup: 1.60->[]
W20260812 08:02:19.440506 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 320) took 549 ms (client timeout 50 ms). Trace:
I20260812 08:02:19.440506 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.61]   Dedup: 1.60->[1.61-1.61]
W20260812 08:02:19.440590 32084 rpcz_store.cc:269] 0812 08:02:18.890790 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:18.890902 (+   112us) service_pool.cc:224] Handling call
0812 08:02:19.440189 (+549287us) raft_consensus.cc:1471] Updating replica for 8 ops
0812 08:02:19.440327 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 59
0812 08:02:19.440333 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:19.440342 (+     9us) raft_consensus.cc:1645] Marking committed up to 59
0812 08:02:19.440346 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.440356 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:19.440470 (+   114us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:19.440494 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262621568}
W20260812 08:02:19.441574 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 826ms of latency in SegmentAllocator::Sync()
W20260812 08:02:20.214672 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9600
W20260812 08:02:20.246992 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 348) took 1032 ms (client timeout 50 ms). Trace:
I20260812 08:02:20.247023 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.247148 31936 rpcz_store.cc:269] 0812 08:02:19.214207 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.214558 (+   351us) service_pool.cc:224] Handling call
0812 08:02:19.214741 (+   183us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:19.214837 (+    96us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:19.214842 (+     5us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:19.214959 (+   117us) write_op.cc:270] Start()
0812 08:02:19.214980 (+    21us) write_op.cc:276] Timestamp: P: 1786521739213408 usec, L: 0
0812 08:02:19.215249 (+   269us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:19.215281 (+    32us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:19.215285 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.215295 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:20.246863 (+1031568us) raft_consensus.cc:1685] finished
0812 08:02:20.246865 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.246960 (+    95us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:19.215247 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:19.215379 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:19.215383 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:19.215385 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:19.215771 (+   386us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:19.215780 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:19.215783 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:19.215960 (+   177us) tablet.cc:701] Row locks acquired
0812 08:02:19.215961 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":774,"thread_start_us":89,"threads_started":1}]]}
I20260812 08:02:20.247396 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.247511 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 349) took 981 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.247565 31942 rpcz_store.cc:269] 0812 08:02:19.265702 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.265827 (+   125us) service_pool.cc:224] Handling call
0812 08:02:20.246979 (+981152us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.247263 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.247269 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.247287 (+    18us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.247294 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.247315 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.247463 (+   148us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:20.247498 (+    35us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255897472}
I20260812 08:02:20.247639 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.247648 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 350) took 619 ms (client timeout 50 ms). Trace:
I20260812 08:02:20.248049 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.248140 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 353) took 132 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.248026 31937 rpcz_store.cc:269] 0812 08:02:19.628493 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.628605 (+   112us) service_pool.cc:224] Handling call
0812 08:02:20.247373 (+618768us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.247494 (+   121us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.247496 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.247502 (+     6us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.247505 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.247513 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.247622 (+   109us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:20.247639 (+    17us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1422635776}
W20260812 08:02:20.248208 31941 rpcz_store.cc:269] 0812 08:02:20.115874 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.115941 (+    67us) service_pool.cc:224] Handling call
0812 08:02:20.247603 (+131662us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.247921 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.247926 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.247935 (+     9us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.247939 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.247950 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.248090 (+   140us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:20.248126 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302377856}
I20260812 08:02:20.248317 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.248347 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 351) took 569 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.248392 31935 rpcz_store.cc:269] 0812 08:02:19.679376 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.679468 (+    92us) service_pool.cc:224] Handling call
0812 08:02:20.248016 (+568548us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.248170 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.248174 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.248187 (+    13us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.248191 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.248202 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.248312 (+   110us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:20.248338 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1306870528}
W20260812 08:02:20.248668 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 352) took 183 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.248798 31939 rpcz_store.cc:269] 0812 08:02:20.065052 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.065185 (+   133us) service_pool.cc:224] Handling call
0812 08:02:20.248288 (+183103us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.248422 (+   134us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.248427 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.248436 (+     9us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.248442 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.248455 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.248555 (+   100us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:20.248580 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420592000}
I20260812 08:02:20.268162 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.268392 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 322) took 985 ms (client timeout 50 ms). Trace:
I20260812 08:02:20.268489 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.268602 32087 rpcz_store.cc:269] 0812 08:02:19.282646 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.282780 (+   134us) service_pool.cc:224] Handling call
0812 08:02:19.440438 (+157658us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:02:19.440641 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:19.440836 (+   195us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:19.440988 (+   152us) write_op.cc:270] Start()
0812 08:02:19.441019 (+    31us) write_op.cc:276] Timestamp: P: 1786521739213408 usec, L: 0
0812 08:02:19.441336 (+   317us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:19.441373 (+    37us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:19.441379 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:19.441389 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:20.268048 (+826659us) raft_consensus.cc:1685] finished
0812 08:02:20.268051 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.268316 (+   265us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:20.268372 (+    56us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:19.441246 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:19.441372 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:19.441376 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:19.441378 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:19.441621 (+   243us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:19.441656 (+    35us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:19.441662 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:19.442511 (+   849us) tablet.cc:701] Row locks acquired
0812 08:02:19.442513 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":361841536,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":27,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":776,"prepare.run_wall_time_us":1330,"row_lock_wait_count":1,"row_lock_wait_us":615,"thread_start_us":87,"threads_started":1}]]}
W20260812 08:02:20.268975 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 323) took 413 ms (client timeout 50 ms). Trace:
I20260812 08:02:20.269095 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.61]   Dedup: 1.61->[]
I20260812 08:02:20.269273 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.61]   Dedup: 1.61->[]
W20260812 08:02:20.269049 32084 rpcz_store.cc:269] 0812 08:02:19.855263 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.855372 (+   109us) service_pool.cc:224] Handling call
0812 08:02:20.268445 (+413073us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:02:20.268674 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.268681 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.268752 (+    71us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.268757 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.268767 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.268926 (+   159us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:20.268956 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":949404928}
W20260812 08:02:20.269517 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 321) took 1038 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.269568 32085 rpcz_store.cc:269] 0812 08:02:19.231386 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.231484 (+    98us) service_pool.cc:224] Handling call
0812 08:02:20.269257 (+1037773us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:02:20.269374 (+   117us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.269376 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.269382 (+     6us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.269385 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.269393 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.269481 (+    88us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:20.269506 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2386067712}
W20260812 08:02:20.269858 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 324) took 363 ms (client timeout 50 ms). Trace:
W20260812 08:02:20.269969 32086 rpcz_store.cc:269] 0812 08:02:19.906419 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:19.906521 (+   102us) service_pool.cc:224] Handling call
0812 08:02:20.269063 (+362542us) raft_consensus.cc:1471] Updating replica for 9 ops
0812 08:02:20.269208 (+   145us) raft_consensus.cc:1534] Early marking committed up to index 60
0812 08:02:20.269210 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:20.269218 (+     8us) raft_consensus.cc:1645] Marking committed up to 60
0812 08:02:20.269221 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.269229 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:20.269821 (+   592us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:20.269846 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833038592}
I20260812 08:02:20.272527   385 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=59, attempt_no=0}) took 21957 ms. Trace:
W20260812 08:02:20.272634 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1268ms of latency in SegmentAllocator::Sync()
I20260812 08:02:20.272697   385 rpcz_store.cc:276] 0812 08:01:58.314998 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:58.315208 (+   210us) service_pool.cc:224] Handling call
0812 08:02:20.272507 (+21957299us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:18.038694 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:18.038805 (+   111us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:18.038810 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:18.038812 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:18.039096 (+   284us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:18.039104 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:18.039107 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:19.213423 (+1174316us) lock_manager.cc:598] Waited 1174057us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:19.213438 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:19.213440 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:19.213486 (+    46us) write_op.cc:270] Start()
0812 08:02:19.213512 (+    26us) write_op.cc:276] Timestamp: P: 1786521739213408 usec, L: 0
0812 08:02:19.213514 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:19.213658 (+   144us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:20.269541 (+1055883us) op_driver.cc:464] REPLICATION: finished
0812 08:02:20.269846 (+   305us) write_op.cc:301] APPLY: starting
0812 08:02:20.269892 (+    46us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:20.269968 (+    76us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:20.269970 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:20.271840 (+  1870us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:20.271899 (+    59us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:20.271905 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:20.272040 (+   135us) log.cc:844] Serialized 319 byte log entry
0812 08:02:20.272131 (+    91us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:20.272164 (+    33us) write_op.cc:454] Released schema lock
0812 08:02:20.272173 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":239,"num_ops":50,"prepare.queue_time_us":19723010,"prepare.run_cpu_time_us":1255,"prepare.run_wall_time_us":1175061,"raft.queue_time_us":107,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":485,"replication_time_us":1055983,"row_lock_wait_count":1,"row_lock_wait_us":1174057,"spinlock_wait_cycles":768,"thread_start_us":82,"threads_started":1}]]}
W20260812 08:02:20.274796 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 975ms of latency in SegmentAllocator::Sync()
W20260812 08:02:20.323969 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 20 similar messages]
W20260812 08:02:20.375020 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:20.502197 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62]   Dedup: 1.61->[1.62-1.62]
W20260812 08:02:20.503396 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 914ms of latency in SegmentAllocator::Sync()
W20260812 08:02:21.250643 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 327) took 977 ms (client timeout 50 ms). Trace:
I20260812 08:02:21.250633 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.61->[1.62-1.62]   Dedup: 1.62->[]
I20260812 08:02:21.251312 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.61->[1.62-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.251375 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 328) took 926 ms (client timeout 50 ms). Trace:
W20260812 08:02:21.251071 32086 rpcz_store.cc:269] 0812 08:02:20.273587 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.273707 (+   120us) service_pool.cc:224] Handling call
0812 08:02:20.273861 (+   154us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:20.273939 (+    78us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:20.273942 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:20.274012 (+    70us) write_op.cc:270] Start()
0812 08:02:20.274036 (+    24us) write_op.cc:276] Timestamp: P: 1786521740272175 usec, L: 0
0812 08:02:20.274337 (+   301us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:20.274371 (+    34us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:20.274375 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.274385 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:21.250424 (+976039us) raft_consensus.cc:1685] finished
0812 08:02:21.250427 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.250620 (+   193us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:20.274411 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:20.274738 (+   327us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:20.274743 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:20.274744 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:20.275078 (+   334us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:20.275090 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:20.275093 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:20.275303 (+   210us) tablet.cc:701] Row locks acquired
0812 08:02:20.275304 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":275,"prepare.run_cpu_time_us":959,"prepare.run_wall_time_us":960,"thread_start_us":102,"threads_started":1}]]}
I20260812 08:02:21.251598 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.61->[1.62-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.251740 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 330) took 411 ms (client timeout 50 ms). Trace:
W20260812 08:02:21.251864 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 329) took 462 ms (client timeout 50 ms). Trace:
W20260812 08:02:21.251863 32087 rpcz_store.cc:269] 0812 08:02:20.840796 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.840883 (+    87us) service_pool.cc:224] Handling call
0812 08:02:21.251285 (+410402us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:21.251441 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.251447 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.251457 (+    10us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.251461 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.251472 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.251694 (+   222us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:21.251727 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943386368}
W20260812 08:02:21.251715 32085 rpcz_store.cc:269] 0812 08:02:20.324992 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.325098 (+   106us) service_pool.cc:224] Handling call
0812 08:02:21.250547 (+925449us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:21.251164 (+   617us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.251169 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.251182 (+    13us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.251186 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.251199 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.251330 (+   131us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:21.251362 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2127724800}
W20260812 08:02:21.251963 32084 rpcz_store.cc:269] 0812 08:02:20.789826 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.790020 (+   194us) service_pool.cc:224] Handling call
0812 08:02:21.251571 (+461551us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:21.251690 (+   119us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.251693 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.251700 (+     7us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.251704 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.251715 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.251830 (+   115us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:21.251853 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060963200}
W20260812 08:02:21.273525 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1340
I20260812 08:02:21.351357 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.61->[1.62-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.418121 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 354) took 916 ms (client timeout 50 ms). Trace:
I20260812 08:02:21.418160 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.418790 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 355) took 866 ms (client timeout 50 ms). Trace:
I20260812 08:02:21.418713 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.418349 31939 rpcz_store.cc:269] 0812 08:02:20.501823 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.501940 (+   117us) service_pool.cc:224] Handling call
0812 08:02:20.502131 (+   191us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:20.502386 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:20.502601 (+   215us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:20.502794 (+   193us) write_op.cc:270] Start()
0812 08:02:20.502820 (+    26us) write_op.cc:276] Timestamp: P: 1786521740272175 usec, L: 0
0812 08:02:20.503143 (+   323us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:20.503185 (+    42us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:20.503191 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:20.503205 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:21.417991 (+914786us) raft_consensus.cc:1685] finished
0812 08:02:21.417993 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.418092 (+    99us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:20.503040 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:20.503174 (+   134us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:20.503178 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:20.503179 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:20.503391 (+   212us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:20.503401 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:20.503404 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:20.504693 (+  1289us) tablet.cc:701] Row locks acquired
0812 08:02:20.504696 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":1085,"prepare.run_wall_time_us":1721,"row_lock_wait_count":1,"row_lock_wait_us":783,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:02:21.418866 31935 rpcz_store.cc:269] 0812 08:02:20.552721 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.552878 (+   157us) service_pool.cc:224] Handling call
0812 08:02:21.418110 (+865232us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:21.418543 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.418546 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.418557 (+    11us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.418560 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.418572 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.418737 (+   165us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:21.418775 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1989291904}
I20260812 08:02:21.419099 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62]   Dedup: 1.62->[]
W20260812 08:02:21.419107 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 357) took 436 ms (client timeout 50 ms). Trace:
W20260812 08:02:21.419188 31937 rpcz_store.cc:269] 0812 08:02:20.982393 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.982527 (+   134us) service_pool.cc:224] Handling call
0812 08:02:21.418675 (+436148us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:21.418922 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.418926 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.418934 (+     8us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.418937 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.418947 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.419071 (+   124us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:21.419096 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002207488}
W20260812 08:02:21.419442 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 356) took 488 ms (client timeout 50 ms). Trace:
W20260812 08:02:21.419503 31941 rpcz_store.cc:269] 0812 08:02:20.931577 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:20.931685 (+   108us) service_pool.cc:224] Handling call
0812 08:02:21.419078 (+487393us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:21.419207 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 61
0812 08:02:21.419210 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:21.419217 (+     7us) raft_consensus.cc:1645] Marking committed up to 61
0812 08:02:21.419221 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.419231 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:21.419398 (+   167us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:21.419429 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119864064}
I20260812 08:02:21.429770 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62]   Dedup: 1.62->[]
I20260812 08:02:21.433240   395 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=60, attempt_no=0}) took 22119 ms. Trace:
I20260812 08:02:21.433424   395 rpcz_store.cc:276] 0812 08:01:59.314199 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:01:59.314327 (+   128us) service_pool.cc:224] Handling call
0812 08:02:21.433221 (+22118894us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:19.213789 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:19.213906 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:19.213910 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:19.213912 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:19.214206 (+   294us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:19.214217 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:19.214222 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:20.272195 (+1057973us) lock_manager.cc:598] Waited 1057674us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:20.272211 (+    16us) tablet.cc:701] Row locks acquired
0812 08:02:20.272213 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:20.272266 (+    53us) write_op.cc:270] Start()
0812 08:02:20.272301 (+    35us) write_op.cc:276] Timestamp: P: 1786521740272175 usec, L: 0
0812 08:02:20.272303 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:20.272465 (+   162us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:21.431101 (+1158636us) op_driver.cc:464] REPLICATION: finished
0812 08:02:21.431518 (+   417us) write_op.cc:301] APPLY: starting
0812 08:02:21.431552 (+    34us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:21.431624 (+    72us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:21.431626 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:21.432897 (+  1271us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:21.432949 (+    52us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:21.432954 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:21.433071 (+   117us) log.cc:844] Serialized 319 byte log entry
0812 08:02:21.433114 (+    43us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:21.433146 (+    32us) write_op.cc:454] Released schema lock
0812 08:02:21.433153 (+     7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":362,"num_ops":50,"prepare.queue_time_us":19898956,"prepare.run_cpu_time_us":1382,"prepare.run_wall_time_us":1058773,"raft.queue_time_us":1086,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":206,"replication_time_us":1158761,"row_lock_wait_count":1,"row_lock_wait_us":1057674,"thread_start_us":81,"threads_started":1}]]}
W20260812 08:02:21.436219 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 942ms of latency in SegmentAllocator::Sync()
W20260812 08:02:21.436542 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1083ms of latency in SegmentAllocator::Sync()
W20260812 08:02:21.536537 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:21.536537 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:21.541388 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1063ms of latency in SegmentAllocator::Sync()
W20260812 08:02:22.379078 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 360) took 943 ms (client timeout 50 ms). Trace:
I20260812 08:02:22.379109 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.379212 31941 rpcz_store.cc:269] 0812 08:02:21.435203 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.435251 (+    48us) service_pool.cc:224] Handling call
0812 08:02:21.435392 (+   141us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:21.435477 (+    85us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:21.435480 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:21.435614 (+   134us) write_op.cc:270] Start()
0812 08:02:21.435640 (+    26us) write_op.cc:276] Timestamp: P: 1786521741433322 usec, L: 0
0812 08:02:21.435956 (+   316us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:21.435990 (+    34us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:21.435995 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.436007 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:22.378929 (+942922us) raft_consensus.cc:1685] finished
0812 08:02:22.378932 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.379032 (+   100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:21.436567 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:21.436704 (+   137us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:21.436709 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:21.436711 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:21.437012 (+   301us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:21.437024 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:21.437028 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:21.437344 (+   316us) tablet.cc:701] Row locks acquired
0812 08:02:21.437346 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":825,"prepare.run_cpu_time_us":850,"prepare.run_wall_time_us":850,"thread_start_us":90,"threads_started":1}]]}
I20260812 08:02:22.379500 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.379577 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 361) took 893 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.379791 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 363) took 447 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.379627 31937 rpcz_store.cc:269] 0812 08:02:21.486503 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.486717 (+   214us) service_pool.cc:224] Handling call
0812 08:02:22.379055 (+892338us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.379349 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.379355 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.379370 (+    15us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.379375 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.379396 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.379531 (+   135us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:22.379562 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051722496}
W20260812 08:02:22.379892 31935 rpcz_store.cc:269] 0812 08:02:21.932323 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.932405 (+    82us) service_pool.cc:224] Handling call
0812 08:02:22.379466 (+447061us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.379604 (+   138us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.379607 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.379615 (+     8us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.379619 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.379629 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.379756 (+   127us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:22.379779 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1027562752}
I20260812 08:02:22.379791 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.380350 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 362) took 499 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.380407 31939 rpcz_store.cc:269] 0812 08:02:21.881251 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.881347 (+    96us) service_pool.cc:224] Handling call
0812 08:02:22.379758 (+498411us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.380176 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.380179 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.380191 (+    12us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.380198 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.380210 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.380313 (+   103us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:22.380341 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1145779072}
W20260812 08:02:22.434474 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561eb1b80
I20260812 08:02:22.483060 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.520116 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 333) took 1085 ms (client timeout 50 ms). Trace:
I20260812 08:02:22.520221 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.520273 32084 rpcz_store.cc:269] 0812 08:02:21.435056 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.435580 (+   524us) service_pool.cc:224] Handling call
0812 08:02:21.435712 (+   132us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:21.435806 (+    94us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:21.435810 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:21.435934 (+   124us) write_op.cc:270] Start()
0812 08:02:21.435959 (+    25us) write_op.cc:276] Timestamp: P: 1786521741433322 usec, L: 0
0812 08:02:21.436283 (+   324us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:21.436324 (+    41us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:21.436329 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:21.436343 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:22.519995 (+1083652us) raft_consensus.cc:1685] finished
0812 08:02:22.519998 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.520087 (+    89us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:21.436834 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:21.436929 (+    95us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:21.436933 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:21.436934 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:21.437218 (+   284us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:21.437228 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:21.437231 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:21.437531 (+   300us) tablet.cc:701] Row locks acquired
0812 08:02:21.437532 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":794,"prepare.run_cpu_time_us":751,"prepare.run_wall_time_us":750,"thread_start_us":77,"threads_started":1}]]}
I20260812 08:02:22.520550 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
I20260812 08:02:22.520838 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.520843 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 336) took 441 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.520958 32086 rpcz_store.cc:269] 0812 08:02:22.079178 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.079282 (+   104us) service_pool.cc:224] Handling call
0812 08:02:22.520520 (+441238us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.520657 (+   137us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.520660 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.520667 (+     7us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.520671 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.520683 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.520806 (+   123us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:22.520831 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1014304640}
I20260812 08:02:22.521129 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63]   Dedup: 1.63->[]
W20260812 08:02:22.520674 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 334) took 1034 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.521165 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 335) took 492 ms (client timeout 50 ms). Trace:
W20260812 08:02:22.521243 32085 rpcz_store.cc:269] 0812 08:02:21.486503 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:21.486681 (+   178us) service_pool.cc:224] Handling call
0812 08:02:22.520168 (+1033487us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.520429 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.520432 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.520442 (+    10us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.520446 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.520456 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.520621 (+   165us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:22.520658 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2376179584}
W20260812 08:02:22.521457 32087 rpcz_store.cc:269] 0812 08:02:22.028326 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.028463 (+   137us) service_pool.cc:224] Handling call
0812 08:02:22.520736 (+492273us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.520982 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 62
0812 08:02:22.520986 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:22.520993 (+     7us) raft_consensus.cc:1645] Marking committed up to 62
0812 08:02:22.520997 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.521007 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:22.521128 (+   121us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:22.521154 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1131699584}
I20260812 08:02:22.525107   403 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=61, attempt_no=0}) took 22173 ms. Trace:
I20260812 08:02:22.525334   403 rpcz_store.cc:276] 0812 08:02:00.351941 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:00.352091 (+   150us) service_pool.cc:224] Handling call
0812 08:02:22.525086 (+22172995us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:20.272588 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:20.272715 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:20.272719 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:20.272721 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:20.273024 (+   303us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:20.273032 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:20.273035 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:21.433301 (+1160266us) lock_manager.cc:598] Waited 1159887us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:21.433316 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:21.433319 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:02:21.433357 (+    38us) write_op.cc:270] Start()
0812 08:02:21.433382 (+    25us) write_op.cc:276] Timestamp: P: 1786521741433322 usec, L: 0
0812 08:02:21.433385 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:21.433540 (+   155us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:22.522497 (+1088957us) op_driver.cc:464] REPLICATION: finished
0812 08:02:22.522759 (+   262us) write_op.cc:301] APPLY: starting
0812 08:02:22.522800 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:22.522870 (+    70us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:22.522873 (+     3us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:22.524424 (+  1551us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:22.524478 (+    54us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:22.524484 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:22.524893 (+   409us) log.cc:844] Serialized 319 byte log entry
0812 08:02:22.524961 (+    68us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:22.524991 (+    30us) write_op.cc:454] Released schema lock
0812 08:02:22.525000 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":50,"prepare.queue_time_us":19920091,"prepare.run_cpu_time_us":1354,"prepare.run_wall_time_us":1161036,"raft.queue_time_us":2268,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":2089,"replication_time_us":1089068,"row_lock_wait_count":1,"row_lock_wait_us":1159887,"thread_start_us":128,"threads_started":1}]]}
W20260812 08:02:22.528299 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 848ms of latency in SegmentAllocator::Sync()
W20260812 08:02:22.529505 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 938ms of latency in SegmentAllocator::Sync()
W20260812 08:02:22.605340 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1083ms of latency in SegmentAllocator::Sync()
W20260812 08:02:22.628595 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:22.628674 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:23.377039 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.377030 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 339) took 849 ms (client timeout 50 ms). Trace:
I20260812 08:02:23.377513 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.377583 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 340) took 799 ms (client timeout 50 ms). Trace:
W20260812 08:02:23.377354 32087 rpcz_store.cc:269] 0812 08:02:22.527338 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.527424 (+    86us) service_pool.cc:224] Handling call
0812 08:02:22.527565 (+   141us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.527632 (+    67us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:22.527634 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:22.527707 (+    73us) write_op.cc:270] Start()
0812 08:02:22.527730 (+    23us) write_op.cc:276] Timestamp: P: 1786521742525885 usec, L: 0
0812 08:02:22.528062 (+   332us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:22.528106 (+    44us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:22.528113 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.528130 (+    17us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:23.376810 (+848680us) raft_consensus.cc:1685] finished
0812 08:02:23.376813 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.376988 (+   175us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:22.528038 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:22.528543 (+   505us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:22.528550 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:22.528553 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:22.528784 (+   231us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:22.528792 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:22.528795 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:22.529001 (+   206us) tablet.cc:701] Row locks acquired
0812 08:02:22.529002 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":231,"prepare.run_cpu_time_us":710,"prepare.run_wall_time_us":1029,"thread_start_us":80,"threads_started":1}]]}
W20260812 08:02:23.377669 32089 rpcz_store.cc:269] 0812 08:02:22.578466 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.578547 (+    81us) service_pool.cc:224] Handling call
0812 08:02:23.376988 (+798441us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.377350 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.377352 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.377362 (+    10us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.377365 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.377376 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.377542 (+   166us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:23.377569 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1835698432}
W20260812 08:02:23.377841 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 341) took 356 ms (client timeout 50 ms). Trace:
I20260812 08:02:23.377954 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.377912 32085 rpcz_store.cc:269] 0812 08:02:23.021357 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.021438 (+    81us) service_pool.cc:224] Handling call
0812 08:02:23.377483 (+356045us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.377645 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.377649 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.377657 (+     8us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.377660 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.377670 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.377801 (+   131us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:23.377828 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818438400}
W20260812 08:02:23.378250 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 342) took 305 ms (client timeout 50 ms). Trace:
W20260812 08:02:23.378315 32086 rpcz_store.cc:269] 0812 08:02:23.072468 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.072592 (+   124us) service_pool.cc:224] Handling call
0812 08:02:23.377927 (+305335us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.378060 (+   133us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.378065 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.378075 (+    10us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.378082 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.378091 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.378210 (+   119us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:23.378237 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701667200}
W20260812 08:02:23.468182 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 366) took 939 ms (client timeout 50 ms). Trace:
I20260812 08:02:23.468231 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.468362 31939 rpcz_store.cc:269] 0812 08:02:22.528385 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.528477 (+    92us) service_pool.cc:224] Handling call
0812 08:02:22.528633 (+   156us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:22.528748 (+   115us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:22.528750 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:22.528888 (+   138us) write_op.cc:270] Start()
0812 08:02:22.528910 (+    22us) write_op.cc:276] Timestamp: P: 1786521742525885 usec, L: 0
0812 08:02:22.529282 (+   372us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:22.529322 (+    40us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:22.529329 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:22.529341 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:23.468039 (+938698us) raft_consensus.cc:1685] finished
0812 08:02:23.468042 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.468148 (+   106us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:22.529263 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:22.529372 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:22.529376 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:22.529378 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:22.529603 (+   225us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:22.529611 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:22.529613 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:22.529791 (+   178us) tablet.cc:701] Row locks acquired
0812 08:02:22.529792 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":278,"prepare.run_cpu_time_us":588,"prepare.run_wall_time_us":588,"thread_start_us":131,"threads_started":1}]]}
I20260812 08:02:23.468617 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.468679 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 367) took 890 ms (client timeout 50 ms). Trace:
W20260812 08:02:23.468730 31935 rpcz_store.cc:269] 0812 08:02:22.578465 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:22.578636 (+   171us) service_pool.cc:224] Handling call
0812 08:02:23.468189 (+889553us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.468460 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.468466 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.468480 (+    14us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.468488 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.468505 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.468629 (+   124us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:23.468666 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2045337344}
W20260812 08:02:23.468946 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 368) took 350 ms (client timeout 50 ms). Trace:
I20260812 08:02:23.469045 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
W20260812 08:02:23.469002 31937 rpcz_store.cc:269] 0812 08:02:23.118109 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.118238 (+   129us) service_pool.cc:224] Handling call
0812 08:02:23.468586 (+350348us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.468746 (+   160us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.468750 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.468757 (+     7us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.468761 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.468771 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.468903 (+   132us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:23.468933 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":805282176}
W20260812 08:02:23.469452 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 369) took 300 ms (client timeout 50 ms). Trace:
W20260812 08:02:23.469540 31941 rpcz_store.cc:269] 0812 08:02:23.168637 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.168801 (+   164us) service_pool.cc:224] Handling call
0812 08:02:23.469010 (+300209us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.469263 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 63
0812 08:02:23.469269 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:23.469279 (+    10us) raft_consensus.cc:1645] Marking committed up to 63
0812 08:02:23.469285 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.469298 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:23.469410 (+   112us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:23.469441 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":689938816}
W20260812 08:02:23.527334 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9080
I20260812 08:02:23.623900 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.64]   Dedup: 1.64->[]
I20260812 08:02:23.693306   409 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=62, attempt_no=0}) took 22236 ms. Trace:
I20260812 08:02:23.693576   409 rpcz_store.cc:276] 0812 08:02:01.456658 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:01.456828 (+   170us) service_pool.cc:224] Handling call
0812 08:02:23.693286 (+22236458us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:21.433654 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:21.433774 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:21.433778 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:21.433780 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:21.434083 (+   303us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:21.434092 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:21.434094 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:22.525865 (+1091771us) lock_manager.cc:598] Waited 1091477us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:22.525882 (+    17us) tablet.cc:701] Row locks acquired
0812 08:02:22.525884 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:22.525930 (+    46us) write_op.cc:270] Start()
0812 08:02:22.525955 (+    25us) write_op.cc:276] Timestamp: P: 1786521742525885 usec, L: 0
0812 08:02:22.525958 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:22.526112 (+   154us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:23.689158 (+1163046us) op_driver.cc:464] REPLICATION: finished
0812 08:02:23.689871 (+   713us) write_op.cc:301] APPLY: starting
0812 08:02:23.689935 (+    64us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:23.690014 (+    79us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:23.690015 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:23.692901 (+  2886us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:23.692956 (+    55us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:23.692963 (+     7us) write_op.cc:312] APPLY: finished
0812 08:02:23.693098 (+   135us) log.cc:844] Serialized 319 byte log entry
0812 08:02:23.693169 (+    71us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:23.693197 (+    28us) write_op.cc:454] Released schema lock
0812 08:02:23.693207 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":636,"num_ops":50,"prepare.queue_time_us":19976247,"prepare.run_cpu_time_us":1287,"prepare.run_wall_time_us":1092551,"raft.queue_time_us":1125,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":292,"replication_time_us":1163135,"row_lock_wait_count":1,"row_lock_wait_us":1091477,"thread_start_us":348,"threads_started":1}]]}
W20260812 08:02:23.695030 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1090ms of latency in SegmentAllocator::Sync()
W20260812 08:02:23.695472 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1013ms of latency in SegmentAllocator::Sync()
I20260812 08:02:23.729210 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65]   Dedup: 1.64->[1.65-1.65]
W20260812 08:02:23.730525 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 979ms of latency in SegmentAllocator::Sync()
W20260812 08:02:23.796458 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:24.695012 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8000
W20260812 08:02:24.709048 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 345) took 1014 ms (client timeout 50 ms). Trace:
I20260812 08:02:24.709105 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.65]   Dedup: 1.65->[]
I20260812 08:02:24.709367 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.65]   Dedup: 1.65->[]
W20260812 08:02:24.709156 32086 rpcz_store.cc:269] 0812 08:02:23.694452 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.694512 (+    60us) service_pool.cc:224] Handling call
0812 08:02:23.694690 (+   178us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:23.694772 (+    82us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:23.694775 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:23.694891 (+   116us) write_op.cc:270] Start()
0812 08:02:23.694911 (+    20us) write_op.cc:276] Timestamp: P: 1786521743693435 usec, L: 0
0812 08:02:23.695220 (+   309us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:23.695272 (+    52us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:23.695278 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.695289 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:24.708931 (+1013642us) raft_consensus.cc:1685] finished
0812 08:02:24.708933 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.709021 (+    88us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:23.695550 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:23.695682 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:23.695688 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:23.695692 (+     4us) tablet.cc:662] Decoding operations
0812 08:02:23.696172 (+   480us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:23.696185 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:23.696188 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:23.696441 (+   253us) tablet.cc:701] Row locks acquired
0812 08:02:23.696442 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":562,"prepare.run_cpu_time_us":963,"prepare.run_wall_time_us":963,"thread_start_us":65,"threads_started":1}]]}
W20260812 08:02:24.709434 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 348) took 409 ms (client timeout 50 ms). Trace:
W20260812 08:02:24.709512 32087 rpcz_store.cc:269] 0812 08:02:24.299676 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.299782 (+   106us) service_pool.cc:224] Handling call
0812 08:02:24.709051 (+409269us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:24.709221 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.709223 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.709233 (+    10us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.709236 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.709245 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.709396 (+   151us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:24.709423 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":940765568}
W20260812 08:02:24.709666 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 346) took 963 ms (client timeout 50 ms). Trace:
W20260812 08:02:24.710000 32085 rpcz_store.cc:269] 0812 08:02:23.746255 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.746342 (+    87us) service_pool.cc:224] Handling call
0812 08:02:24.709336 (+962994us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:24.709476 (+   140us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.709479 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.709486 (+     7us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.709490 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.709500 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.709625 (+   125us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:24.709651 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2213222656}
W20260812 08:02:24.710194 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 370) took 981 ms (client timeout 50 ms). Trace:
I20260812 08:02:24.709678 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.65]   Dedup: 1.65->[]
W20260812 08:02:24.710280 31941 rpcz_store.cc:269] 0812 08:02:23.728796 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.728960 (+   164us) service_pool.cc:224] Handling call
0812 08:02:23.729148 (+   188us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:23.729358 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:23.729694 (+   336us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:23.729825 (+   131us) write_op.cc:270] Start()
0812 08:02:23.729851 (+    26us) write_op.cc:276] Timestamp: P: 1786521743693435 usec, L: 0
0812 08:02:23.730170 (+   319us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:23.730212 (+    42us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:23.730217 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:23.730230 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:24.710032 (+979802us) raft_consensus.cc:1685] finished
0812 08:02:24.710036 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.710171 (+   135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:23.730088 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:23.730215 (+   127us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:23.730219 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:23.730221 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:23.730498 (+   277us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:23.730512 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:23.730514 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:23.731835 (+  1321us) tablet.cc:701] Row locks acquired
0812 08:02:23.731837 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":9,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":998,"prepare.run_wall_time_us":1819,"row_lock_wait_count":1,"row_lock_wait_us":1026,"thread_start_us":94,"threads_started":1}]]}
W20260812 08:02:24.710516 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 347) took 462 ms (client timeout 50 ms). Trace:
W20260812 08:02:24.710557 32089 rpcz_store.cc:269] 0812 08:02:24.248477 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.248655 (+   178us) service_pool.cc:224] Handling call
0812 08:02:24.709647 (+460992us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:24.710346 (+   699us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.710349 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.710356 (+     7us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.710363 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.710374 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.710487 (+   113us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:24.710507 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059796736}
I20260812 08:02:24.710260 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65]   Dedup: 1.65->[]
I20260812 08:02:24.710958 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65]   Dedup: 1.65->[]
W20260812 08:02:24.711024 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 372) took 477 ms (client timeout 50 ms). Trace:
W20260812 08:02:24.711099 31935 rpcz_store.cc:269] 0812 08:02:24.234065 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.234181 (+   116us) service_pool.cc:224] Handling call
0812 08:02:24.710223 (+476042us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:24.710819 (+   596us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.710822 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.710831 (+     9us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.710839 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.710852 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.710986 (+   134us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:24.711013 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1094299904}
I20260812 08:02:24.711201 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65]   Dedup: 1.65->[]
W20260812 08:02:24.711251 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 373) took 425 ms (client timeout 50 ms). Trace:
W20260812 08:02:24.711290 31939 rpcz_store.cc:269] 0812 08:02:24.285554 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.285798 (+   244us) service_pool.cc:224] Handling call
0812 08:02:24.710926 (+425128us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:24.711065 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.711067 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.711073 (+     6us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.711075 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.711083 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.711210 (+   127us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:24.711238 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":976884480}
W20260812 08:02:24.711434 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 371) took 931 ms (client timeout 50 ms). Trace:
I20260812 08:02:24.711476 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.65]   Dedup: 1.65->[]
W20260812 08:02:24.711516 31937 rpcz_store.cc:269] 0812 08:02:23.779877 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:23.779992 (+   115us) service_pool.cc:224] Handling call
0812 08:02:24.711174 (+931182us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:24.711281 (+   107us) raft_consensus.cc:1534] Early marking committed up to index 64
0812 08:02:24.711284 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:24.711290 (+     6us) raft_consensus.cc:1645] Marking committed up to 64
0812 08:02:24.711294 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.711306 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:24.711398 (+    92us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:24.711422 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141045120}
I20260812 08:02:24.774494 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.65]   Dedup: 1.65->[]
I20260812 08:02:24.777910   425 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=63, attempt_no=0}) took 22356 ms. Trace:
I20260812 08:02:24.778134   425 rpcz_store.cc:276] 0812 08:02:02.421248 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:02.421424 (+   176us) service_pool.cc:224] Handling call
0812 08:02:24.777889 (+22356465us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:22.526238 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:22.526404 (+   166us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:22.526408 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:22.526410 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:22.526750 (+   340us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:22.526758 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:22.526762 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:23.693429 (+1166667us) lock_manager.cc:598] Waited 1166328us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:23.693454 (+    25us) tablet.cc:701] Row locks acquired
0812 08:02:23.693457 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:02:23.693511 (+    54us) write_op.cc:270] Start()
0812 08:02:23.693545 (+    34us) write_op.cc:276] Timestamp: P: 1786521743693435 usec, L: 0
0812 08:02:23.693549 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:02:23.693734 (+   185us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:24.775664 (+1081930us) op_driver.cc:464] REPLICATION: finished
0812 08:02:24.775978 (+   314us) write_op.cc:301] APPLY: starting
0812 08:02:24.776013 (+    35us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:24.776074 (+    61us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:24.776076 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:24.777524 (+  1448us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:24.777581 (+    57us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:24.777587 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:24.777719 (+   132us) log.cc:844] Serialized 319 byte log entry
0812 08:02:24.777769 (+    50us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:24.777807 (+    38us) write_op.cc:454] Released schema lock
0812 08:02:24.777815 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":251,"num_ops":50,"prepare.queue_time_us":20104335,"prepare.run_cpu_time_us":1557,"prepare.run_wall_time_us":1167601,"raft.queue_time_us":758,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":403,"replication_time_us":1082073,"row_lock_wait_count":1,"row_lock_wait_us":1166328,"spinlock_wait_cycles":768,"thread_start_us":155,"threads_started":1}]]}
W20260812 08:02:24.780294 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 945ms of latency in SegmentAllocator::Sync()
W20260812 08:02:24.780696 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 902ms of latency in SegmentAllocator::Sync()
W20260812 08:02:24.785569 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1009ms of latency in SegmentAllocator::Sync()
W20260812 08:02:24.880885 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:24.880893 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:25.343889 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 18 similar messages]
W20260812 08:02:25.683516 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 376) took 903 ms (client timeout 50 ms). Trace:
I20260812 08:02:25.683481 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.683640 31942 rpcz_store.cc:269] 0812 08:02:24.779863 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.779904 (+    41us) service_pool.cc:224] Handling call
0812 08:02:24.780030 (+   126us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:24.780101 (+    71us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:24.780105 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:24.780213 (+   108us) write_op.cc:270] Start()
0812 08:02:24.780245 (+    32us) write_op.cc:276] Timestamp: P: 1786521744777850 usec, L: 0
0812 08:02:24.780488 (+   243us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:24.780525 (+    37us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:24.780530 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.780542 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:25.683163 (+902621us) raft_consensus.cc:1685] finished
0812 08:02:25.683165 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.683491 (+   326us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:24.780554 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:24.780654 (+   100us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:24.780658 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:24.780660 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:24.781006 (+   346us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:24.781014 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:24.781016 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:24.781183 (+   167us) tablet.cc:701] Row locks acquired
0812 08:02:24.781184 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":182,"prepare.run_cpu_time_us":730,"prepare.run_wall_time_us":729,"thread_start_us":63,"threads_started":1}]]}
W20260812 08:02:25.683941 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 377) took 853 ms (client timeout 50 ms). Trace:
I20260812 08:02:25.683879 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.684023 31939 rpcz_store.cc:269] 0812 08:02:24.830849 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.831019 (+   170us) service_pool.cc:224] Handling call
0812 08:02:25.683268 (+852249us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.683743 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.683747 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.683758 (+    11us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.683761 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.683773 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.683908 (+   135us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:25.683932 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959398784}
W20260812 08:02:25.684293 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 379) took 339 ms (client timeout 50 ms). Trace:
I20260812 08:02:25.684289 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.684418 31935 rpcz_store.cc:269] 0812 08:02:25.345309 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.345524 (+   215us) service_pool.cc:224] Handling call
0812 08:02:25.683846 (+338322us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.684098 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.684110 (+    12us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.684117 (+     7us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.684123 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.684137 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.684250 (+   113us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:25.684279 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":777556864}
W20260812 08:02:25.684666 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 378) took 390 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.684746 31937 rpcz_store.cc:269] 0812 08:02:25.293984 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.294056 (+    72us) service_pool.cc:224] Handling call
0812 08:02:25.684256 (+390200us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.684484 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.684487 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.684496 (+     9us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.684503 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.684516 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.684627 (+   111us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:25.684654 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":896872320}
I20260812 08:02:25.725909 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.725909 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 351) took 946 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.726137 32089 rpcz_store.cc:269] 0812 08:02:24.779396 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.779499 (+   103us) service_pool.cc:224] Handling call
0812 08:02:24.779639 (+   140us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:24.779718 (+    79us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:24.779720 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:24.779802 (+    82us) write_op.cc:270] Start()
0812 08:02:24.779825 (+    23us) write_op.cc:276] Timestamp: P: 1786521744777850 usec, L: 0
0812 08:02:24.780089 (+   264us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:24.780149 (+    60us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:24.780154 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:24.780165 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:25.725767 (+945602us) raft_consensus.cc:1685] finished
0812 08:02:25.725770 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.725876 (+   106us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:24.780051 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:24.780162 (+   111us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:24.780165 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:24.780167 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:24.780447 (+   280us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:24.780457 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:24.780460 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:24.780695 (+   235us) tablet.cc:701] Row locks acquired
0812 08:02:24.780696 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":711,"prepare.run_wall_time_us":701,"thread_start_us":78,"threads_started":1}]]}
W20260812 08:02:25.726398 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 355) took 158 ms (client timeout 50 ms). Trace:
I20260812 08:02:25.726408 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
I20260812 08:02:25.726794 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.726899 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 352) took 896 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.726465 32084 rpcz_store.cc:269] 0812 08:02:25.567980 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.568082 (+   102us) service_pool.cc:224] Handling call
0812 08:02:25.725857 (+157775us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.726159 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.726163 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.726174 (+    11us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.726178 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.726190 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.726358 (+   168us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:25.726386 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362341120}
I20260812 08:02:25.727053 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.727133 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 353) took 537 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.726964 32087 rpcz_store.cc:269] 0812 08:02:24.830753 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:24.830852 (+    99us) service_pool.cc:224] Handling call
0812 08:02:25.726318 (+895466us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.726683 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.726687 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.726695 (+     8us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.726699 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.726711 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.726842 (+   131us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:25.726880 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058849536}
W20260812 08:02:25.727249 32085 rpcz_store.cc:269] 0812 08:02:25.189300 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.189416 (+   116us) service_pool.cc:224] Handling call
0812 08:02:25.726761 (+537345us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.726920 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.726927 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.726935 (+     8us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.726940 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.726949 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.727092 (+   143us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:25.727120 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235352320}
I20260812 08:02:25.727329 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
W20260812 08:02:25.727416 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 354) took 486 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.727461 32086 rpcz_store.cc:269] 0812 08:02:25.240580 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.240665 (+    85us) service_pool.cc:224] Handling call
0812 08:02:25.727025 (+486360us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.727194 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.727197 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.727204 (+     7us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.727208 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.727229 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.727376 (+   147us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:25.727404 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118049152}
W20260812 08:02:25.727602 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 356) took 108 ms (client timeout 50 ms). Trace:
W20260812 08:02:25.727653 32090 rpcz_store.cc:269] 0812 08:02:25.619049 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.619205 (+   156us) service_pool.cc:224] Handling call
0812 08:02:25.727299 (+108094us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.727424 (+   125us) raft_consensus.cc:1534] Early marking committed up to index 65
0812 08:02:25.727431 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:25.727439 (+     8us) raft_consensus.cc:1645] Marking committed up to 65
0812 08:02:25.727443 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.727452 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:25.727565 (+   113us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:25.727592 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248129920}
W20260812 08:02:25.779141 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce9b80
I20260812 08:02:25.837777 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.66]   Dedup: 1.66->[]
I20260812 08:02:25.840977   435 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=64, attempt_no=0}) took 22183 ms. Trace:
I20260812 08:02:25.841145   435 rpcz_store.cc:276] 0812 08:02:03.657764 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:03.657928 (+   164us) service_pool.cc:224] Handling call
0812 08:02:25.840956 (+22183028us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:23.693885 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:23.694036 (+   151us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:23.694042 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:23.694046 (+     4us) tablet.cc:662] Decoding operations
0812 08:02:23.694514 (+   468us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:23.694526 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:23.694532 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:24.777816 (+1083284us) lock_manager.cc:598] Waited 1083000us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:24.777827 (+    11us) tablet.cc:701] Row locks acquired
0812 08:02:24.777828 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:02:24.777902 (+    74us) write_op.cc:270] Start()
0812 08:02:24.777924 (+    22us) write_op.cc:276] Timestamp: P: 1786521744777850 usec, L: 0
0812 08:02:24.777927 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:24.778075 (+   148us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:25.839053 (+1060978us) op_driver.cc:464] REPLICATION: finished
0812 08:02:25.839356 (+   303us) write_op.cc:301] APPLY: starting
0812 08:02:25.839391 (+    35us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:25.839481 (+    90us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:25.839482 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:25.840612 (+  1130us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:25.840653 (+    41us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:25.840657 (+     4us) write_op.cc:312] APPLY: finished
0812 08:02:25.840757 (+   100us) log.cc:844] Serialized 319 byte log entry
0812 08:02:25.840842 (+    85us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:25.840868 (+    26us) write_op.cc:454] Released schema lock
0812 08:02:25.840876 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":239,"num_ops":50,"prepare.queue_time_us":20035525,"prepare.run_cpu_time_us":1639,"prepare.run_wall_time_us":1084279,"raft.queue_time_us":954,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":355,"replication_time_us":1061081,"row_lock_wait_count":1,"row_lock_wait_us":1083000,"spinlock_wait_cycles":80128,"thread_start_us":90,"threads_started":1}]]}
W20260812 08:02:25.842031 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1024ms of latency in SegmentAllocator::Sync()
W20260812 08:02:25.843402 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 854ms of latency in SegmentAllocator::Sync()
W20260812 08:02:25.944036 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:25.955956 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.67]   Dedup: 1.66->[1.67-1.67]
W20260812 08:02:25.957273 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1026ms of latency in SegmentAllocator::Sync()
I20260812 08:02:26.698424 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.698424 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 382) took 855 ms (client timeout 50 ms). Trace:
W20260812 08:02:26.698848 31937 rpcz_store.cc:269] 0812 08:02:25.842465 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.842536 (+    71us) service_pool.cc:224] Handling call
0812 08:02:25.842692 (+   156us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:25.842762 (+    70us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:25.842764 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:25.842833 (+    69us) write_op.cc:270] Start()
0812 08:02:25.842853 (+    20us) write_op.cc:276] Timestamp: P: 1786521745840990 usec, L: 0
0812 08:02:25.843165 (+   312us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:25.843202 (+    37us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:25.843207 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.843220 (+    13us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:26.698241 (+855021us) raft_consensus.cc:1685] finished
0812 08:02:26.698243 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.698390 (+   147us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:25.843108 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:25.843284 (+   176us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:25.843288 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:25.843290 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:25.843542 (+   252us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:25.843553 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:25.843556 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:25.843736 (+   180us) tablet.cc:701] Row locks acquired
0812 08:02:25.843737 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":683,"thread_start_us":113,"threads_started":1}]]}
W20260812 08:02:26.699105 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 383) took 805 ms (client timeout 50 ms). Trace:
I20260812 08:02:26.699054 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.699162 31935 rpcz_store.cc:269] 0812 08:02:25.893731 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.893877 (+   146us) service_pool.cc:224] Handling call
0812 08:02:26.698370 (+804493us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:26.698891 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.698895 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.698909 (+    14us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.698914 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.698930 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.699059 (+   129us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:26.699092 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1849698048}
I20260812 08:02:26.699352 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.699474 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 384) took 387 ms (client timeout 50 ms). Trace:
W20260812 08:02:26.699538 31939 rpcz_store.cc:269] 0812 08:02:26.311914 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.312032 (+   118us) service_pool.cc:224] Handling call
0812 08:02:26.699005 (+386973us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:26.699223 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.699228 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.699240 (+    12us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.699246 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.699261 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.699417 (+   156us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:26.699455 (+    38us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":889271552}
W20260812 08:02:26.699680 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 385) took 336 ms (client timeout 50 ms). Trace:
W20260812 08:02:26.699889 31942 rpcz_store.cc:269] 0812 08:02:26.363185 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.363284 (+    99us) service_pool.cc:224] Handling call
0812 08:02:26.699323 (+336039us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:26.699471 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.699478 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.699490 (+    12us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.699496 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.699508 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.699633 (+   125us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:26.699664 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":772354432}
W20260812 08:02:26.842249 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8840
I20260812 08:02:26.860006 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.67]   Dedup: 1.67->[]
I20260812 08:02:26.868932   443 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=65, attempt_no=0}) took 22162 ms. Trace:
I20260812 08:02:26.869136   443 rpcz_store.cc:276] 0812 08:02:04.706635 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:04.706963 (+   328us) service_pool.cc:224] Handling call
0812 08:02:26.868912 (+22161949us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:24.778185 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:24.778294 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:24.778297 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:24.778299 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:24.778652 (+   353us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:24.778663 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:24.778669 (+     6us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:25.840991 (+1062322us) lock_manager.cc:598] Waited 1062048us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:25.841012 (+    21us) tablet.cc:701] Row locks acquired
0812 08:02:25.841016 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:02:25.841054 (+    38us) write_op.cc:270] Start()
0812 08:02:25.841078 (+    24us) write_op.cc:276] Timestamp: P: 1786521745840990 usec, L: 0
0812 08:02:25.841082 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:02:25.841226 (+   144us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:26.866714 (+1025488us) op_driver.cc:464] REPLICATION: finished
0812 08:02:26.867132 (+   418us) write_op.cc:301] APPLY: starting
0812 08:02:26.867175 (+    43us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:26.867242 (+    67us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:26.867243 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:26.868544 (+  1301us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:26.868595 (+    51us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:26.868600 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:26.868737 (+   137us) log.cc:844] Serialized 319 byte log entry
0812 08:02:26.868796 (+    59us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:26.868823 (+    27us) write_op.cc:454] Released schema lock
0812 08:02:26.868832 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":285,"num_ops":50,"prepare.queue_time_us":20070832,"prepare.run_cpu_time_us":1325,"prepare.run_wall_time_us":1063120,"raft.queue_time_us":754,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":181,"replication_time_us":1025578,"row_lock_wait_count":1,"row_lock_wait_us":1062048,"thread_start_us":116,"threads_started":1}]]}
W20260812 08:02:26.870141 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1046ms of latency in SegmentAllocator::Sync()
W20260812 08:02:26.871695 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 977ms of latency in SegmentAllocator::Sync()
W20260812 08:02:26.972440 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:26.984055 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 357) took 1028 ms (client timeout 50 ms). Trace:
I20260812 08:02:26.984130 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.984169 32090 rpcz_store.cc:269] 0812 08:02:25.955571 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:25.955700 (+   129us) service_pool.cc:224] Handling call
0812 08:02:25.955895 (+   195us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:25.956098 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:25.956412 (+   314us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:25.956555 (+   143us) write_op.cc:270] Start()
0812 08:02:25.956579 (+    24us) write_op.cc:276] Timestamp: P: 1786521745840990 usec, L: 0
0812 08:02:25.956883 (+   304us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:25.956931 (+    48us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:25.956936 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:25.956951 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:26.983939 (+1026988us) raft_consensus.cc:1685] finished
0812 08:02:26.983942 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.984028 (+    86us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:25.956864 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:25.957021 (+   157us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:25.957026 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:25.957027 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:25.957593 (+   566us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:25.957604 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:25.957607 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:25.958787 (+  1180us) tablet.cc:701] Row locks acquired
0812 08:02:25.958789 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":1168,"prepare.run_wall_time_us":1999,"row_lock_wait_count":1,"row_lock_wait_us":903,"thread_start_us":63,"threads_started":1}]]}
I20260812 08:02:26.984388 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.984460 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 358) took 977 ms (client timeout 50 ms). Trace:
W20260812 08:02:26.984511 32086 rpcz_store.cc:269] 0812 08:02:26.006627 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.006786 (+   159us) service_pool.cc:224] Handling call
0812 08:02:26.984074 (+977288us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:26.984274 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.984277 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.984289 (+    12us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.984292 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.984306 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.984413 (+   107us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:26.984447 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247049472}
I20260812 08:02:26.984691 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.67]   Dedup: 1.67->[]
W20260812 08:02:26.984699 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 359) took 633 ms (client timeout 50 ms). Trace:
W20260812 08:02:26.984862 32085 rpcz_store.cc:269] 0812 08:02:26.350962 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.351070 (+   108us) service_pool.cc:224] Handling call
0812 08:02:26.984360 (+633290us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:26.984540 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.984542 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.984549 (+     7us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.984553 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.984562 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.984667 (+   105us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:26.984688 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455718144}
W20260812 08:02:26.985006 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 360) took 583 ms (client timeout 50 ms). Trace:
I20260812 08:02:26.985033 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.68]   Dedup: 1.67->[1.68-1.68]
W20260812 08:02:26.985054 32087 rpcz_store.cc:269] 0812 08:02:26.401909 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.402017 (+   108us) service_pool.cc:224] Handling call
0812 08:02:26.984659 (+582642us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:26.984824 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 66
0812 08:02:26.984827 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:26.984834 (+     7us) raft_consensus.cc:1645] Marking committed up to 66
0812 08:02:26.984838 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.984850 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:26.984969 (+   119us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:26.984995 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339334912}
W20260812 08:02:26.986537 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1071ms of latency in SegmentAllocator::Sync()
W20260812 08:02:27.073637 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:02:27.849624 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 388) took 979 ms (client timeout 50 ms). Trace:
I20260812 08:02:27.849624 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.68]   Dedup: 1.68->[]
I20260812 08:02:27.850057 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.68]   Dedup: 1.68->[]
W20260812 08:02:27.850165 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 389) took 928 ms (client timeout 50 ms). Trace:
W20260812 08:02:27.849815 31942 rpcz_store.cc:269] 0812 08:02:26.870463 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.870659 (+   196us) service_pool.cc:224] Handling call
0812 08:02:26.870858 (+   199us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:26.870936 (+    78us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:26.870939 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:26.871045 (+   106us) write_op.cc:270] Start()
0812 08:02:26.871066 (+    21us) write_op.cc:276] Timestamp: P: 1786521746868887 usec, L: 0
0812 08:02:26.871462 (+   396us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:26.871535 (+    73us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:26.871542 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.871553 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:27.849478 (+977925us) raft_consensus.cc:1685] finished
0812 08:02:27.849482 (+     4us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:27.849600 (+   118us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:26.871386 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:26.871517 (+   131us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:26.871520 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:26.871521 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:26.871826 (+   305us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:26.871837 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:26.871839 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:26.872122 (+   283us) tablet.cc:701] Row locks acquired
0812 08:02:26.872125 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":244,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":819,"thread_start_us":110,"threads_started":1}]]}
W20260812 08:02:27.850250 31939 rpcz_store.cc:269] 0812 08:02:26.922138 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.922230 (+    92us) service_pool.cc:224] Handling call
0812 08:02:27.849573 (+927343us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:27.849871 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:27.849875 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:27.849886 (+    11us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:27.849890 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:27.849902 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:27.850111 (+   209us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:27.850147 (+    36us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132233088}
I20260812 08:02:27.850492 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.68]   Dedup: 1.68->[]
W20260812 08:02:27.850545 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 391) took 356 ms (client timeout 50 ms). Trace:
W20260812 08:02:27.850628 31937 rpcz_store.cc:269] 0812 08:02:27.493938 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.494053 (+   115us) service_pool.cc:224] Handling call
0812 08:02:27.849998 (+355945us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:27.850205 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:27.850281 (+    76us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:27.850295 (+    14us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:27.850303 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:27.850320 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:27.850471 (+   151us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:27.850511 (+    40us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818092800}
W20260812 08:02:27.850835 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 390) took 407 ms (client timeout 50 ms). Trace:
W20260812 08:02:27.850957 31935 rpcz_store.cc:269] 0812 08:02:27.443182 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.443280 (+    98us) service_pool.cc:224] Handling call
0812 08:02:27.850456 (+407176us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:27.850632 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:27.850637 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:27.850648 (+    11us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:27.850656 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:27.850668 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:27.850787 (+   119us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:27.850820 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":935981440}
W20260812 08:02:27.870131 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce8580
I20260812 08:02:27.880650 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.68]   Dedup: 1.68->[]
I20260812 08:02:27.919787   455 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=66, attempt_no=0}) took 22190 ms. Trace:
I20260812 08:02:27.919952   455 rpcz_store.cc:276] 0812 08:02:05.729225 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:05.729321 (+    96us) service_pool.cc:224] Handling call
0812 08:02:27.919765 (+22190444us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:25.841331 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:25.841445 (+   114us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:25.841448 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:25.841450 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:25.841796 (+   346us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:25.841805 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:25.841808 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:26.868900 (+1027092us) lock_manager.cc:598] Waited 1026822us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:26.868915 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:26.868918 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:02:26.868962 (+    44us) write_op.cc:270] Start()
0812 08:02:26.868988 (+    26us) write_op.cc:276] Timestamp: P: 1786521746868887 usec, L: 0
0812 08:02:26.868990 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:26.869143 (+   153us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:27.917079 (+1047936us) op_driver.cc:464] REPLICATION: finished
0812 08:02:27.917541 (+   462us) write_op.cc:301] APPLY: starting
0812 08:02:27.917646 (+   105us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:27.917703 (+    57us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:27.917704 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:27.919232 (+  1528us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:27.919294 (+    62us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:27.919299 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:27.919437 (+   138us) log.cc:844] Serialized 319 byte log entry
0812 08:02:27.919526 (+    89us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:27.919690 (+   164us) write_op.cc:454] Released schema lock
0812 08:02:27.919699 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":371,"num_ops":50,"prepare.queue_time_us":20111473,"prepare.run_cpu_time_us":1392,"prepare.run_wall_time_us":1027875,"raft.queue_time_us":1565,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":1047982,"row_lock_wait_count":1,"row_lock_wait_us":1026822,"spinlock_wait_cycles":5760,"thread_start_us":128,"threads_started":1}]]}
W20260812 08:02:27.920267 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1170ms of latency in SegmentAllocator::Sync()
W20260812 08:02:27.921828 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1076ms of latency in SegmentAllocator::Sync()
W20260812 08:02:28.022411 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:28.058257 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.68]   Dedup: 1.68->[]
W20260812 08:02:28.058329 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 361) took 1086 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.058383 32084 rpcz_store.cc:269] 0812 08:02:26.972227 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:26.972391 (+   164us) service_pool.cc:224] Handling call
0812 08:02:26.984966 (+ 12575us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:26.985157 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:26.985383 (+   226us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:26.985526 (+   143us) write_op.cc:270] Start()
0812 08:02:26.985552 (+    26us) write_op.cc:276] Timestamp: P: 1786521746868887 usec, L: 0
0812 08:02:26.986002 (+   450us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:26.986083 (+    81us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:26.986091 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:26.986107 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:28.058022 (+1071915us) raft_consensus.cc:1685] finished
0812 08:02:28.058025 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.058244 (+   219us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:28.058305 (+    61us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:26.985833 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:26.985940 (+   107us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:26.985944 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:26.985945 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:26.986657 (+   712us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:26.986720 (+    63us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:26.986727 (+     7us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:26.987731 (+  1004us) tablet.cc:701] Row locks acquired
0812 08:02:26.987733 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":28290816,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":8,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":1285,"prepare.run_wall_time_us":1986,"row_lock_wait_count":1,"row_lock_wait_us":784,"spinlock_wait_cycles":42240,"thread_start_us":144,"threads_started":1}]]}
I20260812 08:02:28.058573 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.68]   Dedup: 1.68->[]
I20260812 08:02:28.058935 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.68]   Dedup: 1.68->[]
W20260812 08:02:28.058998 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 363) took 554 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.059041 32085 rpcz_store.cc:269] 0812 08:02:27.504950 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.505120 (+   170us) service_pool.cc:224] Handling call
0812 08:02:28.058525 (+553405us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:28.058796 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:28.058801 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.058809 (+     8us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:28.058815 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.058828 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.058946 (+   118us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:28.058983 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1271777152}
W20260812 08:02:28.059288 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 362) took 1035 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.059360 32087 rpcz_store.cc:269] 0812 08:02:27.023528 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.023765 (+   237us) service_pool.cc:224] Handling call
0812 08:02:28.058905 (+1035140us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:28.059085 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:28.059088 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.059095 (+     7us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:28.059098 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.059109 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.059247 (+   138us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:28.059278 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379522304}
I20260812 08:02:28.059599 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.68->[1.69-1.69]
W20260812 08:02:28.060314 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 364) took 504 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.060385 32086 rpcz_store.cc:269] 0812 08:02:27.555580 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.555684 (+   104us) service_pool.cc:224] Handling call
0812 08:02:28.058202 (+502518us) raft_consensus.cc:1471] Updating replica for 3 ops
0812 08:02:28.058420 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 67
0812 08:02:28.058423 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.058434 (+    11us) raft_consensus.cc:1645] Marking committed up to 67
0812 08:02:28.058438 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.058452 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.060267 (+  1815us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:28.060299 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155250944}
W20260812 08:02:28.060555 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1162ms of latency in SegmentAllocator::Sync()
W20260812 08:02:28.920882 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561ce82c0
W20260812 08:02:28.998409 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 394) took 1077 ms (client timeout 50 ms). Trace:
I20260812 08:02:28.998425 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69]   Dedup: 1.69->[]
W20260812 08:02:28.998533 31935 rpcz_store.cc:269] 0812 08:02:27.920894 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.920938 (+    44us) service_pool.cc:224] Handling call
0812 08:02:27.921091 (+   153us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:27.921180 (+    89us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:27.921182 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:27.921306 (+   124us) write_op.cc:270] Start()
0812 08:02:27.921327 (+    21us) write_op.cc:276] Timestamp: P: 1786521747919553 usec, L: 0
0812 08:02:27.921629 (+   302us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:27.921669 (+    40us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:27.921674 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:27.921686 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:28.998264 (+1076578us) raft_consensus.cc:1685] finished
0812 08:02:28.998267 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.998376 (+   109us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:27.921586 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:27.921712 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:27.921716 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:27.921718 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:27.922008 (+   290us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:27.922019 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:27.922022 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:27.922349 (+   327us) tablet.cc:701] Row locks acquired
0812 08:02:27.922350 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":193,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":826,"thread_start_us":99,"threads_started":1}]]}
I20260812 08:02:28.998932 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69]   Dedup: 1.69->[]
W20260812 08:02:28.999008 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 395) took 1026 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.999063 31937 rpcz_store.cc:269] 0812 08:02:27.972197 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:27.972375 (+   178us) service_pool.cc:224] Handling call
0812 08:02:28.998384 (+1026009us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:28.998804 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.998807 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.998818 (+    11us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.998821 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.998831 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.998960 (+   129us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:28.998994 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2359018368}
W20260812 08:02:28.999182 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 397) took 605 ms (client timeout 50 ms). Trace:
I20260812 08:02:28.999222 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69]   Dedup: 1.69->[]
W20260812 08:02:28.999279 31942 rpcz_store.cc:269] 0812 08:02:28.393525 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.393715 (+   190us) service_pool.cc:224] Handling call
0812 08:02:28.998900 (+605185us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:28.999026 (+   126us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.999029 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.999035 (+     6us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.999039 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.999047 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.999153 (+   106us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:28.999172 (+    19us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391338880}
I20260812 08:02:28.999425 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69]   Dedup: 1.69->[]
W20260812 08:02:28.999500 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 396) took 656 ms (client timeout 50 ms). Trace:
I20260812 08:02:28.999619 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69]   Dedup: 1.69->[]
W20260812 08:02:28.999682 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 398) took 92 ms (client timeout 50 ms). Trace:
W20260812 08:02:28.999822 31941 rpcz_store.cc:269] 0812 08:02:28.906800 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.906916 (+   116us) service_pool.cc:224] Handling call
0812 08:02:28.999402 (+ 92486us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:28.999511 (+   109us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.999515 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.999523 (+     8us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.999528 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.999541 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.999649 (+   108us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:28.999672 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211362176}
W20260812 08:02:28.999620 31939 rpcz_store.cc:269] 0812 08:02:28.342732 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.342800 (+    68us) service_pool.cc:224] Handling call
0812 08:02:28.999188 (+656388us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:28.999321 (+   133us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.999324 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.999333 (+     9us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.999340 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.999353 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.999460 (+   107us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:28.999487 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1509102336}
W20260812 08:02:28.999977 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 399) took 42 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.000031 31936 rpcz_store.cc:269] 0812 08:02:28.957627 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.957785 (+   158us) service_pool.cc:224] Handling call
0812 08:02:28.999591 (+ 41806us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:28.999818 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.999821 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:28.999829 (+     8us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.999836 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.999847 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:28.999945 (+    98us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:28.999967 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95679872}
I20260812 08:02:29.093343   461 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=67, attempt_no=0}) took 22339 ms. Trace:
W20260812 08:02:29.093739 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1161ms of latency in SegmentAllocator::Sync()
I20260812 08:02:29.093848   461 rpcz_store.cc:276] 0812 08:02:06.754207 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:06.754287 (+    80us) service_pool.cc:224] Handling call
0812 08:02:29.093320 (+22339033us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:26.869249 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:26.869369 (+   120us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:26.869373 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:26.869374 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:26.869668 (+   294us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:26.869676 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:26.869680 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:27.919560 (+1049880us) lock_manager.cc:598] Waited 1049600us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:27.919573 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:27.919575 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:27.919621 (+    46us) write_op.cc:270] Start()
0812 08:02:27.919642 (+    21us) write_op.cc:276] Timestamp: P: 1786521747919553 usec, L: 0
0812 08:02:27.919644 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:27.919784 (+   140us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:29.091170 (+1171386us) op_driver.cc:464] REPLICATION: finished
0812 08:02:29.091430 (+   260us) write_op.cc:301] APPLY: starting
0812 08:02:29.091465 (+    35us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:29.091517 (+    52us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:29.091518 (+     1us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:29.092722 (+  1204us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:29.092773 (+    51us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:29.092778 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:29.092899 (+   121us) log.cc:844] Serialized 319 byte log entry
0812 08:02:29.093189 (+   290us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:29.093225 (+    36us) write_op.cc:454] Released schema lock
0812 08:02:29.093235 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":191,"num_ops":50,"prepare.queue_time_us":20114487,"prepare.run_cpu_time_us":1268,"prepare.run_wall_time_us":1050626,"raft.queue_time_us":1233,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":1167,"replication_time_us":1171477,"row_lock_wait_count":1,"row_lock_wait_us":1049600,"thread_start_us":97,"threads_started":1}]]}
W20260812 08:02:29.095302 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1242ms of latency in SegmentAllocator::Sync()
W20260812 08:02:29.195756 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:29.223270 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.69->[]
W20260812 08:02:29.223460 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 365) took 1198 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.223799 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 368) took 563 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.223590 32090 rpcz_store.cc:269] 0812 08:02:28.025514 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.025679 (+   165us) service_pool.cc:224] Handling call
0812 08:02:28.059530 (+ 33851us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:28.059713 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:28.059885 (+   172us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:28.059964 (+    79us) write_op.cc:270] Start()
0812 08:02:28.059992 (+    28us) write_op.cc:276] Timestamp: P: 1786521747919553 usec, L: 0
0812 08:02:28.060328 (+   336us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:28.060361 (+    33us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:28.060366 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:28.060376 (+    10us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:29.223018 (+1162642us) raft_consensus.cc:1685] finished
0812 08:02:29.223021 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.223259 (+   238us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.223430 (+   171us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:28.060296 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:28.060420 (+   124us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:28.060424 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:28.060425 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:28.060745 (+   320us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:28.060755 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:28.060757 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:28.061861 (+  1104us) tablet.cc:701] Row locks acquired
0812 08:02:28.061862 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":76451584,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":228,"prepare.run_cpu_time_us":811,"prepare.run_wall_time_us":1632,"row_lock_wait_count":1,"row_lock_wait_us":900,"thread_start_us":81,"threads_started":1}]]}
I20260812 08:02:29.223840 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.69->[]
I20260812 08:02:29.224129 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.69->[]
W20260812 08:02:29.224216 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 367) took 614 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.224269 32087 rpcz_store.cc:269] 0812 08:02:28.609449 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.609758 (+   309us) service_pool.cc:224] Handling call
0812 08:02:29.223800 (+614042us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:29.223993 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:29.223996 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:29.224007 (+    11us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:29.224017 (+    10us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.224026 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.224169 (+   143us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.224199 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1411460480}
I20260812 08:02:29.224370 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.69->[]
W20260812 08:02:29.224421 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 370) took 117 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.224658 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 369) took 168 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.224464 32089 rpcz_store.cc:269] 0812 08:02:29.107416 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.107512 (+    96us) service_pool.cc:224] Handling call
0812 08:02:29.224096 (+116584us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:29.224251 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:29.224254 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:29.224261 (+     7us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:29.224266 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.224279 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.224386 (+   107us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.224409 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267549440}
W20260812 08:02:29.224725 32084 rpcz_store.cc:269] 0812 08:02:29.055824 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.055942 (+   118us) service_pool.cc:224] Handling call
0812 08:02:29.224340 (+168398us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:29.224482 (+   142us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:29.224487 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:29.224496 (+     9us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:29.224502 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.224516 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.224625 (+   109us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.224647 (+    22us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":386732928}
I20260812 08:02:29.224831 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.69]   Dedup: 1.69->[]
W20260812 08:02:29.223853 32085 rpcz_store.cc:269] 0812 08:02:28.660223 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.660381 (+   158us) service_pool.cc:224] Handling call
0812 08:02:29.223131 (+562750us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:29.223612 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:29.223616 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:29.223628 (+    12us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:29.223631 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.223643 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.223761 (+   118us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.223789 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1293598464}
W20260812 08:02:29.225067 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 366) took 1148 ms (client timeout 50 ms). Trace:
W20260812 08:02:29.225129 32086 rpcz_store.cc:269] 0812 08:02:28.076858 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:28.076986 (+   128us) service_pool.cc:224] Handling call
0812 08:02:29.224807 (+1147821us) raft_consensus.cc:1471] Updating replica for 4 ops
0812 08:02:29.224911 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 68
0812 08:02:29.224916 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:29.224923 (+     7us) raft_consensus.cc:1645] Marking committed up to 68
0812 08:02:29.224926 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.224937 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:29.225032 (+    95us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:29.225055 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2639222016}
I20260812 08:02:29.499639 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.70]   Dedup: 1.69->[1.70-1.70]
W20260812 08:02:29.501221 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 850ms of latency in SegmentAllocator::Sync()
W20260812 08:02:29.600597 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 11: this message will repeat every 5th retry.
W20260812 08:02:30.094648 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18840
I20260812 08:02:30.337910 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.337903 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 401) took 1243 ms (client timeout 50 ms). Trace:
I20260812 08:02:30.338256 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.338397 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 402) took 1193 ms (client timeout 50 ms). Trace:
I20260812 08:02:30.338557 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.338114 31936 rpcz_store.cc:269] 0812 08:02:29.094414 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.094463 (+    49us) service_pool.cc:224] Handling call
0812 08:02:29.094625 (+   162us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:29.094701 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:29.094704 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:29.094788 (+    84us) write_op.cc:270] Start()
0812 08:02:29.094813 (+    25us) write_op.cc:276] Timestamp: P: 1786521749093308 usec, L: 0
0812 08:02:29.095110 (+   297us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:29.095145 (+    35us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:29.095152 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.095166 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:30.337761 (+1242595us) raft_consensus.cc:1685] finished
0812 08:02:30.337764 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.337866 (+   102us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:29.095117 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:29.095226 (+   109us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:29.095232 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:29.095236 (+     4us) tablet.cc:662] Decoding operations
0812 08:02:29.095557 (+   321us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:29.095566 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:29.095568 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:29.095788 (+   220us) tablet.cc:701] Row locks acquired
0812 08:02:29.095789 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":726,"thread_start_us":71,"threads_started":1}]]}
W20260812 08:02:30.338549 31939 rpcz_store.cc:269] 0812 08:02:29.145370 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.145474 (+   104us) service_pool.cc:224] Handling call
0812 08:02:30.337869 (+1192395us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.338127 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.338134 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.338147 (+    13us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.338151 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.338171 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.338348 (+   177us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:30.338381 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2741764992}
I20260812 08:02:30.339005 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.338622 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 406) took 312 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.339100 31935 rpcz_store.cc:269] 0812 08:02:30.026192 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.026315 (+   123us) service_pool.cc:224] Handling call
0812 08:02:30.338230 (+311915us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.338400 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.338407 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.338417 (+    10us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.338424 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.338435 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.338548 (+   113us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:30.338573 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":716857088}
W20260812 08:02:30.339087 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 403) took 832 ms (client timeout 50 ms). Trace:
I20260812 08:02:30.339439 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.339565 31941 rpcz_store.cc:269] 0812 08:02:29.506315 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.506456 (+   141us) service_pool.cc:224] Handling call
0812 08:02:30.338521 (+832065us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.338905 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.338910 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.338919 (+     9us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.338922 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.338932 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.339047 (+   115us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:30.339074 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1913017728}
W20260812 08:02:30.339781 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 404) took 782 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.339855 31942 rpcz_store.cc:269] 0812 08:02:29.557827 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.557929 (+   102us) service_pool.cc:224] Handling call
0812 08:02:30.338977 (+781048us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.339096 (+   119us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.339102 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.339111 (+     9us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.339118 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.339131 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.339735 (+   604us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:30.339768 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1795758208}
W20260812 08:02:30.340003 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 405) took 365 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.340081 31937 rpcz_store.cc:269] 0812 08:02:29.974919 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.975024 (+   105us) service_pool.cc:224] Handling call
0812 08:02:30.339413 (+364389us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.339778 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.339785 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.339797 (+    12us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.339812 (+    15us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.339826 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.339955 (+   129us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:30.339987 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":837416832}
I20260812 08:02:30.351826 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.351826 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 371) took 852 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.352020 32086 rpcz_store.cc:269] 0812 08:02:29.499222 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.499357 (+   135us) service_pool.cc:224] Handling call
0812 08:02:29.499580 (+   223us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:29.499854 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:29.500094 (+   240us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:29.500223 (+   129us) write_op.cc:270] Start()
0812 08:02:29.500251 (+    28us) write_op.cc:276] Timestamp: P: 1786521749093308 usec, L: 0
0812 08:02:29.500785 (+   534us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:29.500837 (+    52us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:29.500843 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:29.500857 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:30.351677 (+850820us) raft_consensus.cc:1685] finished
0812 08:02:30.351680 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.351799 (+   119us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:29.500732 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:29.500869 (+   137us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:29.500876 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:29.500879 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:29.501198 (+   319us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:29.501215 (+    17us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:29.501218 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:29.502113 (+   895us) tablet.cc:701] Row locks acquired
0812 08:02:29.502115 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":397,"prepare.run_cpu_time_us":909,"prepare.run_wall_time_us":1451,"row_lock_wait_count":1,"row_lock_wait_us":597,"thread_start_us":244,"threads_started":1}]]}
W20260812 08:02:30.352258 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 372) took 801 ms (client timeout 50 ms). Trace:
I20260812 08:02:30.352221 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.352303 32085 rpcz_store.cc:269] 0812 08:02:29.550426 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:29.550529 (+   103us) service_pool.cc:224] Handling call
0812 08:02:30.351774 (+801245us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:30.352051 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.352054 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.352066 (+    12us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.352070 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.352082 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.352218 (+   136us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:30.352246 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1842031488}
I20260812 08:02:30.352476 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.352583 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 374) took 253 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.352641 32089 rpcz_store.cc:269] 0812 08:02:30.099433 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.099529 (+    96us) service_pool.cc:224] Handling call
0812 08:02:30.352184 (+252655us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:30.352363 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.352368 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.352379 (+    11us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.352386 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.352399 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.352534 (+   135us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:30.352567 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580519680}
W20260812 08:02:30.352752 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 373) took 304 ms (client timeout 50 ms). Trace:
W20260812 08:02:30.352811 32084 rpcz_store.cc:269] 0812 08:02:30.048487 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.048614 (+   127us) service_pool.cc:224] Handling call
0812 08:02:30.352450 (+303836us) raft_consensus.cc:1471] Updating replica for 5 ops
0812 08:02:30.352568 (+   118us) raft_consensus.cc:1534] Early marking committed up to index 69
0812 08:02:30.352573 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:30.352583 (+    10us) raft_consensus.cc:1645] Marking committed up to 69
0812 08:02:30.352589 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.352598 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:30.352708 (+   110us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:30.352739 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":698203904}
I20260812 08:02:30.530632 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.70]   Dedup: 1.70->[]
W20260812 08:02:30.534320 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 976ms of latency in SegmentAllocator::Sync()
I20260812 08:02:30.534631   469 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=68, attempt_no=0}) took 22719 ms. Trace:
I20260812 08:02:30.534746   469 rpcz_store.cc:276] 0812 08:02:07.814757 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:07.814937 (+   180us) service_pool.cc:224] Handling call
0812 08:02:30.534605 (+22719668us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:27.919896 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:27.920015 (+   119us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:27.920019 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:27.920021 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:27.920338 (+   317us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:27.920345 (+     7us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:27.920348 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:29.093267 (+1172919us) lock_manager.cc:598] Waited 1172628us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:29.093282 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:29.093284 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:29.093342 (+    58us) write_op.cc:270] Start()
0812 08:02:29.093370 (+    28us) write_op.cc:276] Timestamp: P: 1786521749093308 usec, L: 0
0812 08:02:29.093373 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:29.093525 (+   152us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:30.531843 (+1438318us) op_driver.cc:464] REPLICATION: finished
0812 08:02:30.532181 (+   338us) write_op.cc:301] APPLY: starting
0812 08:02:30.532220 (+    39us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:30.532290 (+    70us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:30.532292 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:30.533632 (+  1340us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:30.533681 (+    49us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:30.533686 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:30.533829 (+   143us) log.cc:844] Serialized 319 byte log entry
0812 08:02:30.533905 (+    76us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:30.534501 (+   596us) write_op.cc:454] Released schema lock
0812 08:02:30.534511 (+    10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":283,"num_ops":50,"prepare.queue_time_us":20104197,"prepare.run_cpu_time_us":1452,"prepare.run_wall_time_us":1173717,"raft.queue_time_us":381,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"replication_time_us":1438340,"row_lock_wait_count":1,"row_lock_wait_us":1172628,"spinlock_wait_cycles":4864,"thread_start_us":94,"threads_started":1}]]}
W20260812 08:02:30.535851 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1102ms of latency in SegmentAllocator::Sync()
W20260812 08:02:30.584671 32165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT) [suppressed 20 similar messages]
I20260812 08:02:30.585691 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.71]   Dedup: 1.70->[1.71-1.71]
W20260812 08:02:30.586786 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1013ms of latency in SegmentAllocator::Sync()
W20260812 08:02:30.635533 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:31.535163 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18b00
I20260812 08:02:31.600450 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.600452 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 407) took 1015 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.600867 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 408) took 964 ms (client timeout 50 ms). Trace:
I20260812 08:02:31.600841 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.600713 31937 rpcz_store.cc:269] 0812 08:02:30.585378 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.585456 (+    78us) service_pool.cc:224] Handling call
0812 08:02:30.585637 (+   181us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:30.585873 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:30.586054 (+   181us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:30.586135 (+    81us) write_op.cc:270] Start()
0812 08:02:30.586159 (+    24us) write_op.cc:276] Timestamp: P: 1786521750533932 usec, L: 0
0812 08:02:30.586467 (+   308us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:30.586511 (+    44us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:30.586517 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.586528 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:31.600288 (+1013760us) raft_consensus.cc:1685] finished
0812 08:02:31.600290 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.600428 (+   138us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:30.586383 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:30.586509 (+   126us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:30.586513 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:30.586515 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:30.586843 (+   328us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:30.586852 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:30.586855 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:30.587999 (+  1144us) tablet.cc:701] Row locks acquired
0812 08:02:30.588001 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":900,"prepare.run_wall_time_us":1690,"row_lock_wait_count":1,"row_lock_wait_us":883,"thread_start_us":84,"threads_started":1}]]}
W20260812 08:02:31.600935 31942 rpcz_store.cc:269] 0812 08:02:30.636290 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.636366 (+    76us) service_pool.cc:224] Handling call
0812 08:02:31.600398 (+964032us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:31.600684 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.600687 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.600700 (+    13us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.600705 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.600718 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.600826 (+   108us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:31.600852 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2216616320}
W20260812 08:02:31.601447 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 410) took 503 ms (client timeout 50 ms). Trace:
I20260812 08:02:31.601266 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.601523 31935 rpcz_store.cc:269] 0812 08:02:31.098109 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.098213 (+   104us) service_pool.cc:224] Handling call
0812 08:02:31.600798 (+502585us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:31.601104 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.601107 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.601116 (+     9us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.601120 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.601131 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.601398 (+   267us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:31.601428 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155363200}
I20260812 08:02:31.601820 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.601824 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 409) took 554 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.601977 31941 rpcz_store.cc:269] 0812 08:02:31.046968 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.047095 (+   127us) service_pool.cc:224] Handling call
0812 08:02:31.601240 (+554145us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:31.601638 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.601643 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.601651 (+     8us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.601658 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.601673 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.601784 (+   111us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:31.601811 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1273882496}
I20260812 08:02:31.605352   478 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=69, attempt_no=0}) took 22774 ms. Trace:
I20260812 08:02:31.605453   478 rpcz_store.cc:276] 0812 08:02:08.830979 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:08.831064 (+    85us) service_pool.cc:224] Handling call
0812 08:02:31.605334 (+22774270us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:29.093643 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:29.093762 (+   119us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:29.093766 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:29.093768 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:29.094126 (+   358us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:29.094134 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:29.094138 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:30.533937 (+1439799us) lock_manager.cc:598] Waited 1439425us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:30.533948 (+    11us) tablet.cc:701] Row locks acquired
0812 08:02:30.533950 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:30.533994 (+    44us) write_op.cc:270] Start()
0812 08:02:30.534017 (+    23us) write_op.cc:276] Timestamp: P: 1786521750533932 usec, L: 0
0812 08:02:30.534019 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:30.534159 (+   140us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:31.602895 (+1068736us) op_driver.cc:464] REPLICATION: finished
0812 08:02:31.603117 (+   222us) write_op.cc:301] APPLY: starting
0812 08:02:31.603167 (+    50us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:31.603248 (+    81us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:31.603252 (+     4us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:31.604866 (+  1614us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:31.604925 (+    59us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:31.604932 (+     7us) write_op.cc:312] APPLY: finished
0812 08:02:31.605066 (+   134us) log.cc:844] Serialized 319 byte log entry
0812 08:02:31.605200 (+   134us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:31.605237 (+    37us) write_op.cc:454] Released schema lock
0812 08:02:31.605246 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":159,"num_ops":50,"prepare.queue_time_us":20262206,"prepare.run_cpu_time_us":1412,"prepare.run_wall_time_us":1440624,"raft.queue_time_us":318,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"replication_time_us":1068841,"row_lock_wait_count":1,"row_lock_wait_us":1439425,"thread_start_us":107,"threads_started":1}]]}
W20260812 08:02:31.605736 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 958ms of latency in SegmentAllocator::Sync()
W20260812 08:02:31.607633 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1030ms of latency in SegmentAllocator::Sync()
I20260812 08:02:31.638447 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.638448 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 377) took 1103 ms (client timeout 50 ms). Trace:
I20260812 08:02:31.638823 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.638870 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 379) took 702 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.638659 32084 rpcz_store.cc:269] 0812 08:02:30.534780 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.534883 (+   103us) service_pool.cc:224] Handling call
0812 08:02:30.535016 (+   133us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:30.535208 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:30.535211 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:30.535299 (+    88us) write_op.cc:270] Start()
0812 08:02:30.535320 (+    21us) write_op.cc:276] Timestamp: P: 1786521750533932 usec, L: 0
0812 08:02:30.535675 (+   355us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:30.535959 (+   284us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:30.535970 (+    11us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:30.535986 (+    16us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:31.638290 (+1102304us) raft_consensus.cc:1685] finished
0812 08:02:31.638293 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.638423 (+   130us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:30.535611 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:30.535743 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:30.535747 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:02:30.535748 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:30.536111 (+   363us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:30.536120 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:30.536123 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:30.536312 (+   189us) tablet.cc:701] Row locks acquired
0812 08:02:30.536313 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":209,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":759,"thread_start_us":139,"threads_started":1}]]}
W20260812 08:02:31.639055 32085 rpcz_store.cc:269] 0812 08:02:30.936634 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.936738 (+   104us) service_pool.cc:224] Handling call
0812 08:02:31.638395 (+701657us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.638677 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.638680 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.638692 (+    12us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.638696 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.638707 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.638833 (+   126us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:31.638858 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612707200}
I20260812 08:02:31.639245 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.639293 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 378) took 1053 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.639340 32089 rpcz_store.cc:269] 0812 08:02:30.585518 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.585597 (+    79us) service_pool.cc:224] Handling call
0812 08:02:31.638777 (+1053180us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.639003 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.639007 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.639016 (+     9us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.639020 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.639051 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.639247 (+   196us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:31.639280 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2421569792}
W20260812 08:02:31.639514 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 380) took 651 ms (client timeout 50 ms). Trace:
I20260812 08:02:31.639521 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.639592 32086 rpcz_store.cc:269] 0812 08:02:30.988464 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:30.988641 (+   177us) service_pool.cc:224] Handling call
0812 08:02:31.639215 (+650574us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.639343 (+   128us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.639350 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.639359 (+     9us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.639363 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.639372 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.639482 (+   110us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:31.639503 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1495693568}
I20260812 08:02:31.639935 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.71]   Dedup: 1.71->[]
W20260812 08:02:31.640015 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 382) took 105 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.640098 32090 rpcz_store.cc:269] 0812 08:02:31.534497 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.534685 (+   188us) service_pool.cc:224] Handling call
0812 08:02:31.639492 (+104807us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.639812 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.639815 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.639821 (+     6us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.639824 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.639860 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.639979 (+   119us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:31.640003 (+    24us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240601856}
W20260812 08:02:31.640239 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 381) took 156 ms (client timeout 50 ms). Trace:
W20260812 08:02:31.640290 32087 rpcz_store.cc:269] 0812 08:02:31.483343 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.483485 (+   142us) service_pool.cc:224] Handling call
0812 08:02:31.639912 (+156427us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.640051 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 70
0812 08:02:31.640057 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:31.640066 (+     9us) raft_consensus.cc:1645] Marking committed up to 70
0812 08:02:31.640070 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.640080 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:31.640194 (+   114us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:31.640223 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":359201536}
W20260812 08:02:31.707581 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:32.087353 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.71->[1.72-1.72]
W20260812 08:02:32.088789 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1033ms of latency in SegmentAllocator::Sync()
W20260812 08:02:32.606559 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d18dc0
I20260812 08:02:32.638275 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
W20260812 08:02:32.638330 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 413) took 1031 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.638440 31939 rpcz_store.cc:269] 0812 08:02:31.606391 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.606436 (+    45us) service_pool.cc:224] Handling call
0812 08:02:31.606758 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:31.606834 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:31.606836 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:31.606911 (+    75us) write_op.cc:270] Start()
0812 08:02:31.606933 (+    22us) write_op.cc:276] Timestamp: P: 1786521751605286 usec, L: 0
0812 08:02:31.607383 (+   450us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:31.607424 (+    41us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:31.607429 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:31.607440 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:32.638032 (+1030592us) raft_consensus.cc:1685] finished
0812 08:02:32.638035 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.638252 (+   217us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:31.607312 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:31.607420 (+   108us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:31.607423 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:31.607425 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:31.607669 (+   244us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:31.607677 (+     8us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:31.607679 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:31.607951 (+   272us) tablet.cc:701] Row locks acquired
0812 08:02:31.607953 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":302,"prepare.run_cpu_time_us":700,"prepare.run_wall_time_us":700,"thread_start_us":219,"threads_started":1}]]}
I20260812 08:02:32.638711 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
W20260812 08:02:32.638746 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 416) took 497 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.639000 31942 rpcz_store.cc:269] 0812 08:02:32.141395 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.141488 (+    93us) service_pool.cc:224] Handling call
0812 08:02:32.638223 (+496735us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:32.638471 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.638474 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:32.638486 (+    12us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.638490 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.638503 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.638701 (+   198us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:32.638731 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141922176}
I20260812 08:02:32.639096 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
I20260812 08:02:32.639276 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
W20260812 08:02:32.639359 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 417) took 122 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.639434 31937 rpcz_store.cc:269] 0812 08:02:32.517399 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.517529 (+   130us) service_pool.cc:224] Handling call
0812 08:02:32.639079 (+121550us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:32.639183 (+   104us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.639185 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:32.639191 (+     6us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.639194 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.639201 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.639316 (+   115us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:32.639347 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279120256}
I20260812 08:02:32.639531 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
W20260812 08:02:32.639648 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 414) took 982 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.639752 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 415) took 550 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.639766 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 418) took 71 ms (client timeout 50 ms). Trace:
W20260812 08:02:32.639827 31935 rpcz_store.cc:269] 0812 08:02:32.089639 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.089688 (+    49us) service_pool.cc:224] Handling call
0812 08:02:32.638583 (+548895us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:32.639005 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.639011 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:32.639021 (+    10us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.639028 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.639042 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.639713 (+   671us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:32.639743 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261884672}
W20260812 08:02:32.639736 31941 rpcz_store.cc:269] 0812 08:02:31.657396 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:31.657529 (+   133us) service_pool.cc:224] Handling call
0812 08:02:32.639250 (+981721us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:32.639428 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.639433 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:32.639443 (+    10us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.639447 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.639458 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.639584 (+   126us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:32.639616 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2257278208}
W20260812 08:02:32.639833 31936 rpcz_store.cc:269] 0812 08:02:32.568494 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.568628 (+   134us) service_pool.cc:224] Handling call
0812 08:02:32.639509 (+ 70881us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:32.639614 (+   105us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.639617 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:32.639625 (+     8us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.639630 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.639644 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:32.639743 (+    99us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:32.639756 (+    13us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162604928}
I20260812 08:02:33.070909 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.074688 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 971ms of latency in SegmentAllocator::Sync()
I20260812 08:02:33.075244   485 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=70, attempt_no=0}) took 23160 ms. Trace:
I20260812 08:02:33.075593   485 rpcz_store.cc:276] 0812 08:02:09.914752 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:09.914834 (+    82us) service_pool.cc:224] Handling call
0812 08:02:33.075229 (+23160395us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:30.534303 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:30.534415 (+   112us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:30.534421 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:30.534424 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:30.534763 (+   339us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:30.534774 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:30.534777 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:31.605268 (+1070491us) lock_manager.cc:598] Waited 1070212us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:31.605283 (+    15us) tablet.cc:701] Row locks acquired
0812 08:02:31.605286 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:02:31.605356 (+    70us) write_op.cc:270] Start()
0812 08:02:31.605379 (+    23us) write_op.cc:276] Timestamp: P: 1786521751605286 usec, L: 0
0812 08:02:31.605380 (+     1us) op_driver.cc:348] REPLICATION: starting
0812 08:02:31.605546 (+   166us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:33.072140 (+1466594us) op_driver.cc:464] REPLICATION: finished
0812 08:02:33.072414 (+   274us) write_op.cc:301] APPLY: starting
0812 08:02:33.072452 (+    38us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:33.072517 (+    65us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:33.072519 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:33.073913 (+  1394us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:33.073961 (+    48us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:33.073966 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:33.074073 (+   107us) log.cc:844] Serialized 319 byte log entry
0812 08:02:33.074172 (+    99us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:33.075130 (+   958us) write_op.cc:454] Released schema lock
0812 08:02:33.075138 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":208,"num_ops":50,"prepare.queue_time_us":20619055,"prepare.run_cpu_time_us":1341,"prepare.run_wall_time_us":1071328,"raft.queue_time_us":512,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"replication_time_us":1466720,"row_lock_wait_count":1,"row_lock_wait_us":1070212,"spinlock_wait_cycles":40576,"thread_start_us":121,"threads_started":1}]]}
W20260812 08:02:33.076849 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 1059ms of latency in SegmentAllocator::Sync()
W20260812 08:02:33.122426 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 383) took 1035 ms (client timeout 50 ms). Trace:
I20260812 08:02:33.122473 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.122612 32087 rpcz_store.cc:269] 0812 08:02:32.086993 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.087087 (+    94us) service_pool.cc:224] Handling call
0812 08:02:32.087290 (+   203us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:32.087545 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:32.087879 (+   334us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:32.088083 (+   204us) write_op.cc:270] Start()
0812 08:02:32.088113 (+    30us) write_op.cc:276] Timestamp: P: 1786521751605286 usec, L: 0
0812 08:02:32.088510 (+   397us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:32.088555 (+    45us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:32.088561 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:32.088573 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:33.122295 (+1033722us) raft_consensus.cc:1685] finished
0812 08:02:33.122298 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.122393 (+    95us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:32.088405 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:32.088520 (+   115us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:32.088527 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:32.088529 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:32.088788 (+   259us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:32.088797 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:32.088799 (+     2us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:32.090173 (+  1374us) tablet.cc:701] Row locks acquired
0812 08:02:32.090176 (+     3us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":211,"prepare.run_cpu_time_us":954,"prepare.run_wall_time_us":1842,"row_lock_wait_count":1,"row_lock_wait_us":1019,"thread_start_us":151,"threads_started":1}]]}
I20260812 08:02:33.123014 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.123136 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 385) took 579 ms (client timeout 50 ms). Trace:
W20260812 08:02:33.123485 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 386) took 529 ms (client timeout 50 ms). Trace:
I20260812 08:02:33.123239 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.123558 32086 rpcz_store.cc:269] 0812 08:02:32.543319 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.543405 (+    86us) service_pool.cc:224] Handling call
0812 08:02:33.122421 (+579016us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:33.122746 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:33.122752 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:33.122768 (+    16us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:33.122775 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.122891 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.123067 (+   176us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:33.123113 (+    46us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331152768}
W20260812 08:02:33.123580 32089 rpcz_store.cc:269] 0812 08:02:32.594270 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.594372 (+   102us) service_pool.cc:224] Handling call
0812 08:02:33.122983 (+528611us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:33.123122 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:33.123126 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:33.123134 (+     8us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:33.123144 (+    10us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.123155 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.123292 (+   137us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:33.123326 (+    34us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1215192320}
I20260812 08:02:33.123857 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.123895 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 387) took 95 ms (client timeout 50 ms). Trace:
W20260812 08:02:33.123948 32085 rpcz_store.cc:269] 0812 08:02:33.028537 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.028674 (+   137us) service_pool.cc:224] Handling call
0812 08:02:33.123208 (+ 94534us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:33.123627 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:33.123631 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:33.123640 (+     9us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:33.123643 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.123725 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.123853 (+   128us) spinlock_profiling.cc:243] Waited 94.3 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:33.123880 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216954624}
I20260812 08:02:33.124094 32090 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72]   Dedup: 1.72->[]
W20260812 08:02:33.124243 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 388) took 44 ms (client timeout 50 ms). Trace:
W20260812 08:02:33.124351 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 384) took 986 ms (client timeout 50 ms). Trace:
W20260812 08:02:33.124325 32084 rpcz_store.cc:269] 0812 08:02:33.079788 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.079951 (+   163us) service_pool.cc:224] Handling call
0812 08:02:33.123819 (+ 43868us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:33.123967 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:33.123971 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:33.123980 (+     9us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:33.123988 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.124001 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.124118 (+   117us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:33.124228 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100258176}
W20260812 08:02:33.124414 32090 rpcz_store.cc:269] 0812 08:02:32.138070 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:32.138198 (+   128us) service_pool.cc:224] Handling call
0812 08:02:33.124062 (+985864us) raft_consensus.cc:1471] Updating replica for 2 ops
0812 08:02:33.124198 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 71
0812 08:02:33.124201 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:33.124208 (+     7us) raft_consensus.cc:1645] Marking committed up to 71
0812 08:02:33.124211 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.124221 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:33.124322 (+   101us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:33.124343 (+    21us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266757376}
W20260812 08:02:33.129936 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1140ms of latency in SegmentAllocator::Sync()
W20260812 08:02:33.177947 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:33.227981 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:34.075448 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d182c0
I20260812 08:02:34.136379 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.136387 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 421) took 1060 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.136776 31935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 423) took 501 ms (client timeout 50 ms). Trace:
I20260812 08:02:34.136749 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.136837 31935 rpcz_store.cc:269] 0812 08:02:33.635346 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.635449 (+   103us) service_pool.cc:224] Handling call
0812 08:02:34.136330 (+500881us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.136531 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.136533 (+     2us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.136543 (+    10us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.136546 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.136554 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.136731 (+   177us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:34.136762 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1151422720}
I20260812 08:02:34.136973 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.137037 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 425) took 113 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.137082 31942 rpcz_store.cc:269] 0812 08:02:34.023206 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.023355 (+   149us) service_pool.cc:224] Handling call
0812 08:02:34.136724 (+113369us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.136872 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.136875 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.136881 (+     6us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.136885 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.136894 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.137008 (+   114us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:34.137028 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":260249856}
I20260812 08:02:34.137162 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.137284 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 422) took 1009 ms (client timeout 50 ms). Trace:
I20260812 08:02:34.137367 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.137341 31936 rpcz_store.cc:269] 0812 08:02:33.127891 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.127970 (+    79us) service_pool.cc:224] Handling call
0812 08:02:34.136942 (+1008972us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.137064 (+   122us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.137070 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.137079 (+     9us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.137086 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.137099 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.137246 (+   147us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:34.137271 (+    25us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2320033024}
W20260812 08:02:34.137401 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 426) took 63 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.137565 31939 rpcz_store.cc:269] 0812 08:02:34.073971 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.074068 (+    97us) service_pool.cc:224] Handling call
0812 08:02:34.137143 (+ 63075us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.137251 (+   108us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.137254 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.137260 (+     6us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.137263 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.137270 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.137372 (+   102us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:34.137392 (+    20us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144619904}
W20260812 08:02:34.136646 31941 rpcz_store.cc:269] 0812 08:02:33.075669 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.075757 (+    88us) service_pool.cc:224] Handling call
0812 08:02:33.075921 (+   164us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:33.076008 (+    87us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:33.076012 (+     4us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:33.076227 (+   215us) write_op.cc:270] Start()
0812 08:02:33.076274 (+    47us) write_op.cc:276] Timestamp: P: 1786521753074211 usec, L: 0
0812 08:02:33.076651 (+   377us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:33.076966 (+   315us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:33.076973 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.076988 (+    15us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:34.136241 (+1059253us) raft_consensus.cc:1685] finished
0812 08:02:34.136244 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.136338 (+    94us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:33.076572 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:33.076762 (+   190us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:33.076769 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:33.076772 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:33.077221 (+   449us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:33.077230 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:33.077233 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:33.077627 (+   394us) tablet.cc:701] Row locks acquired
0812 08:02:33.077629 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":197,"prepare.run_cpu_time_us":936,"prepare.run_wall_time_us":1123,"thread_start_us":112,"threads_started":1}]]}
W20260812 08:02:34.137686 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 424) took 451 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.137846 31937 rpcz_store.cc:269] 0812 08:02:33.686105 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.686228 (+   123us) service_pool.cc:224] Handling call
0812 08:02:34.137341 (+451113us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.137496 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.137502 (+     6us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.137512 (+    10us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.137518 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.137532 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.137642 (+   110us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:34.137671 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036499840}
W20260812 08:02:34.270841 32090 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 389) took 1144 ms (client timeout 50 ms). Trace:
I20260812 08:02:34.270874 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.270983 32090 rpcz_store.cc:269] 0812 08:02:33.127593 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.127794 (+   201us) service_pool.cc:224] Handling call
0812 08:02:33.127989 (+   195us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:33.128088 (+    99us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:33.128997 (+   909us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:33.129251 (+   254us) write_op.cc:270] Start()
0812 08:02:33.129286 (+    35us) write_op.cc:276] Timestamp: P: 1786521753074211 usec, L: 0
0812 08:02:33.129663 (+   377us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:33.129702 (+    39us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:33.129708 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:33.129722 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:34.270567 (+1140845us) raft_consensus.cc:1685] finished
0812 08:02:34.270570 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.270806 (+   236us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:33.129642 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:33.129774 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:33.129791 (+    17us) write_op.cc:435] Acquired schema lock
0812 08:02:33.129792 (+     1us) tablet.cc:662] Decoding operations
0812 08:02:33.130277 (+   485us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:33.130371 (+    94us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:33.130464 (+    93us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:33.131519 (+  1055us) tablet.cc:701] Row locks acquired
0812 08:02:33.131521 (+     2us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":8,"prepare.queue_time_us":269,"prepare.run_cpu_time_us":1310,"prepare.run_wall_time_us":1954,"row_lock_wait_count":1,"row_lock_wait_us":722,"thread_start_us":117,"threads_started":1}]]}
I20260812 08:02:34.271239 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.271302 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 390) took 1093 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.271378 32084 rpcz_store.cc:269] 0812 08:02:33.177689 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.177828 (+   139us) service_pool.cc:224] Handling call
0812 08:02:34.270814 (+1092986us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.271067 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.271071 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.271089 (+    18us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.271098 (+     9us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.271117 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.271232 (+   115us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:34.271283 (+    51us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2513128832}
W20260812 08:02:34.271540 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 392) took 663 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.271602 32086 rpcz_store.cc:269] 0812 08:02:33.608439 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.608579 (+   140us) service_pool.cc:224] Handling call
0812 08:02:34.271204 (+662625us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.271337 (+   133us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.271340 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.271347 (+     7us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.271351 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.271362 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.271495 (+   133us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:34.271526 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1523456896}
I20260812 08:02:34.271800 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.272065 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 393) took 258 ms (client timeout 50 ms). Trace:
I20260812 08:02:34.272099 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.272133 32089 rpcz_store.cc:269] 0812 08:02:34.014126 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.014260 (+   134us) service_pool.cc:224] Handling call
0812 08:02:34.271770 (+257510us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.271887 (+   117us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.271890 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.271897 (+     7us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.271901 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.271912 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.272027 (+   115us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:34.272053 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":591808256}
I20260812 08:02:34.272445 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.272532 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 394) took 207 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.272598 32087 rpcz_store.cc:269] 0812 08:02:34.065365 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.065487 (+   122us) service_pool.cc:224] Handling call
0812 08:02:34.272027 (+206540us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.272219 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.272224 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.272234 (+    10us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.272240 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.272251 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.272477 (+   226us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:34.272514 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474521472}
W20260812 08:02:34.272751 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 391) took 715 ms (client timeout 50 ms). Trace:
W20260812 08:02:34.272816 32085 rpcz_store.cc:269] 0812 08:02:33.557232 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:33.557322 (+    90us) service_pool.cc:224] Handling call
0812 08:02:34.272416 (+715094us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.272544 (+   128us) raft_consensus.cc:1534] Early marking committed up to index 72
0812 08:02:34.272549 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:34.272556 (+     7us) raft_consensus.cc:1645] Marking committed up to 72
0812 08:02:34.272562 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.272573 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:34.272707 (+   134us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:34.272736 (+    29us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1644066176}
I20260812 08:02:34.567809 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
I20260812 08:02:34.568145 32085 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73]   Dedup: 1.73->[]
W20260812 08:02:34.573345 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 1072ms of latency in SegmentAllocator::Sync()
I20260812 08:02:34.573633   494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=71, attempt_no=0}) took 23756 ms. Trace:
W20260812 08:02:34.575281 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 875ms of latency in SegmentAllocator::Sync()
W20260812 08:02:34.575357 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 1058ms of latency in SegmentAllocator::Sync()
I20260812 08:02:34.573765   494 rpcz_store.cc:276] 0812 08:02:10.817212 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:10.817316 (+   104us) service_pool.cc:224] Handling call
0812 08:02:34.573619 (+23756303us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:31.605660 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:31.605792 (+   132us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:31.605799 (+     7us) write_op.cc:435] Acquired schema lock
0812 08:02:31.605802 (+     3us) tablet.cc:662] Decoding operations
0812 08:02:31.606114 (+   312us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:31.606123 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:31.606127 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:33.074208 (+1468081us) lock_manager.cc:598] Waited 1467766us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:33.074221 (+    13us) tablet.cc:701] Row locks acquired
0812 08:02:33.074222 (+     1us) write_op.cc:260] PREPARE: finished
0812 08:02:33.074273 (+    51us) write_op.cc:270] Start()
0812 08:02:33.074298 (+    25us) write_op.cc:276] Timestamp: P: 1786521753074211 usec, L: 0
0812 08:02:33.074301 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:33.074461 (+   160us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:34.569250 (+1494789us) op_driver.cc:464] REPLICATION: finished
0812 08:02:34.570189 (+   939us) write_op.cc:301] APPLY: starting
0812 08:02:34.570215 (+    26us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:34.570262 (+    47us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:34.570264 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:34.572658 (+  2394us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:34.572706 (+    48us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:34.572711 (+     5us) write_op.cc:312] APPLY: finished
0812 08:02:34.572824 (+   113us) log.cc:844] Serialized 319 byte log entry
0812 08:02:34.572886 (+    62us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:34.573536 (+   650us) write_op.cc:454] Released schema lock
0812 08:02:34.573545 (+     9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":878,"num_ops":50,"prepare.queue_time_us":20787937,"prepare.run_cpu_time_us":1552,"prepare.run_wall_time_us":1468866,"replication_time_us":1494908,"row_lock_wait_count":1,"row_lock_wait_us":1467766,"thread_start_us":72,"threads_started":1}]]}
W20260812 08:02:34.676000 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:34.676000 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:35.451153 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.451170 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 429) took 877 ms (client timeout 50 ms). Trace:
I20260812 08:02:35.451515 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.451604 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 430) took 825 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.451433 31937 rpcz_store.cc:269] 0812 08:02:34.573862 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.573921 (+    59us) service_pool.cc:224] Handling call
0812 08:02:34.574064 (+   143us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.574148 (+    84us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:34.574154 (+     6us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:34.574373 (+   219us) write_op.cc:270] Start()
0812 08:02:34.574396 (+    23us) write_op.cc:276] Timestamp: P: 1786521754572868 usec, L: 0
0812 08:02:34.575017 (+   621us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:34.575054 (+    37us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:34.575060 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.575072 (+    12us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:35.450979 (+875907us) raft_consensus.cc:1685] finished
0812 08:02:35.450982 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.451138 (+   156us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:34.574935 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:34.575032 (+    97us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:34.575038 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:02:34.575040 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:34.575348 (+   308us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:34.575362 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:34.575365 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:34.575652 (+   287us) tablet.cc:701] Row locks acquired
0812 08:02:34.575653 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":400,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":792,"thread_start_us":107,"threads_started":1}]]}
I20260812 08:02:35.451741 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.451870 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 431) took 369 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.451745 31939 rpcz_store.cc:269] 0812 08:02:34.625802 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.625869 (+    67us) service_pool.cc:224] Handling call
0812 08:02:35.451101 (+825232us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.451333 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.451336 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.451345 (+     9us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.451348 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.451358 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.451538 (+   180us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:35.451571 (+    33us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1897503488}
W20260812 08:02:35.452041 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 432) took 318 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.451998 31941 rpcz_store.cc:269] 0812 08:02:35.082353 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.082420 (+    67us) service_pool.cc:224] Handling call
0812 08:02:35.451485 (+369065us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.451617 (+   132us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.451620 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.451628 (+     8us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.451631 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.451640 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.451826 (+   186us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:35.451857 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848353664}
W20260812 08:02:35.452165 31936 rpcz_store.cc:269] 0812 08:02:35.133212 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.133347 (+   135us) service_pool.cc:224] Handling call
0812 08:02:35.451711 (+318364us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.451857 (+   146us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.451864 (+     7us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.451873 (+     9us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.451881 (+     8us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.451896 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.452004 (+   108us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:35.452030 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731712896}
W20260812 08:02:35.574275 32334 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 11785448468237383701) tablet 049a7ef1624e4420952a5c0960ea2492 cur holder 0x557561d19080
I20260812 08:02:35.604453 31941 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.634016 32085 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 397) took 1060 ms (client timeout 50 ms). Trace:
I20260812 08:02:35.634073 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.634217 32085 rpcz_store.cc:269] 0812 08:02:34.573866 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.573923 (+    57us) service_pool.cc:224] Handling call
0812 08:02:34.574062 (+   139us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:34.574132 (+    70us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:34.574134 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:34.574321 (+   187us) write_op.cc:270] Start()
0812 08:02:34.574347 (+    26us) write_op.cc:276] Timestamp: P: 1786521754572868 usec, L: 0
0812 08:02:34.574970 (+   623us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:34.575020 (+    50us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:34.575026 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:34.575040 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:35.633888 (+1058848us) raft_consensus.cc:1685] finished
0812 08:02:35.633890 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.633983 (+    93us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:34.574644 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:34.574917 (+   273us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:34.574922 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:34.574924 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:34.575338 (+   414us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:34.575351 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:34.575355 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:34.575735 (+   380us) tablet.cc:701] Row locks acquired
0812 08:02:34.575739 (+     4us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":6,"prepare.queue_time_us":218,"prepare.run_cpu_time_us":1167,"prepare.run_wall_time_us":1167,"thread_start_us":119,"threads_started":1}]]}
I20260812 08:02:35.634624 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.634677 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 398) took 1008 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.634744 32087 rpcz_store.cc:269] 0812 08:02:34.625731 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:34.625809 (+    78us) service_pool.cc:224] Handling call
0812 08:02:35.634021 (+1008212us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.634461 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.634464 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.634475 (+    11us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.634477 (+     2us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.634487 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.634629 (+   142us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:35.634661 (+    32us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2318188288}
I20260812 08:02:35.634936 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.635006 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 399) took 630 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.635074 32089 rpcz_store.cc:269] 0812 08:02:35.004606 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.004880 (+   274us) service_pool.cc:224] Handling call
0812 08:02:35.634561 (+629681us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.634711 (+   150us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.634714 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.634721 (+     7us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.634724 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.634733 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.634964 (+   231us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:35.634994 (+    30us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1447557888}
I20260812 08:02:35.635182 32086 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.74]   Dedup: 1.74->[]
W20260812 08:02:35.635213 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 400) took 579 ms (client timeout 50 ms). Trace:
W20260812 08:02:35.635278 32084 rpcz_store.cc:269] 0812 08:02:35.055429 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.055545 (+   116us) service_pool.cc:224] Handling call
0812 08:02:35.634907 (+579362us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.635042 (+   135us) raft_consensus.cc:1534] Early marking committed up to index 73
0812 08:02:35.635045 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:35.635052 (+     7us) raft_consensus.cc:1645] Marking committed up to 73
0812 08:02:35.635059 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.635068 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:35.635174 (+   106us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:35.635200 (+    26us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331885824}
I20260812 08:02:35.638957   502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=72, attempt_no=0}) took 23834 ms. Trace:
I20260812 08:02:35.639114   502 rpcz_store.cc:276] 0812 08:02:11.804250 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:11.804340 (+    90us) service_pool.cc:224] Handling call
0812 08:02:35.638937 (+23834597us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:33.074561 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:33.074737 (+   176us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:33.074742 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:33.074744 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:33.075053 (+   309us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:33.075064 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:33.075068 (+     4us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:34.572920 (+1497852us) lock_manager.cc:598] Waited 1497567us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:34.572930 (+    10us) tablet.cc:701] Row locks acquired
0812 08:02:34.572932 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:34.572976 (+    44us) write_op.cc:270] Start()
0812 08:02:34.572997 (+    21us) write_op.cc:276] Timestamp: P: 1786521754572868 usec, L: 0
0812 08:02:34.573000 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:02:34.573137 (+   137us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:35.636163 (+1063026us) op_driver.cc:464] REPLICATION: finished
0812 08:02:35.636374 (+   211us) write_op.cc:301] APPLY: starting
0812 08:02:35.636408 (+    34us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:35.636472 (+    64us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:35.636474 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:35.637834 (+  1360us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:35.637875 (+    41us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:35.637878 (+     3us) write_op.cc:312] APPLY: finished
0812 08:02:35.638016 (+   138us) log.cc:844] Serialized 319 byte log entry
0812 08:02:35.638066 (+    50us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:35.638843 (+   777us) write_op.cc:454] Released schema lock
0812 08:02:35.638855 (+    12us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":151,"num_ops":50,"prepare.queue_time_us":21269715,"prepare.run_cpu_time_us":1336,"prepare.run_wall_time_us":1498715,"raft.queue_time_us":218,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":2313,"replication_time_us":1063128,"row_lock_wait_count":1,"row_lock_wait_us":1497567,"spinlock_wait_cycles":4864,"thread_start_us":66,"threads_started":1}]]}
W20260812 08:02:35.640537 32329 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475: Injecting 907ms of latency in SegmentAllocator::Sync()
W20260812 08:02:35.641544 32330 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69: Injecting 978ms of latency in SegmentAllocator::Sync()
W20260812 08:02:35.646411 32317 log.cc:510] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290: Injecting 882ms of latency in SegmentAllocator::Sync()
W20260812 08:02:35.689182 32164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT) [suppressed 20 similar messages]
W20260812 08:02:35.740581 32164 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer 37575efb04814de79c4ef62b1a472a69 (127.22.208.129:37473): Couldn't send request to peer 37575efb04814de79c4ef62b1a472a69. Status: Timed out: UpdateConsensus RPC to 127.22.208.129:37473 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:35.740581 32165 consensus_peers.cc:597] T 049a7ef1624e4420952a5c0960ea2492 P dba6326e5e2b4938880eac572a3f5290 -> Peer ddcf1211ef9942bf910d906bbbe02475 (127.22.208.130:45345): Couldn't send request to peer ddcf1211ef9942bf910d906bbbe02475. Status: Timed out: UpdateConsensus RPC to 127.22.208.130:45345 timed out after 0.050s (SENT). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:36.548231 32086 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 403) took 909 ms (client timeout 50 ms). Trace:
I20260812 08:02:36.548378 32087 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.548365 32086 rpcz_store.cc:269] 0812 08:02:35.638791 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.638889 (+    98us) service_pool.cc:224] Handling call
0812 08:02:35.639058 (+   169us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.639132 (+    74us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:35.639135 (+     3us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:35.639480 (+   345us) write_op.cc:270] Start()
0812 08:02:35.639530 (+    50us) write_op.cc:276] Timestamp: P: 1786521755638074 usec, L: 0
0812 08:02:35.639951 (+   421us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:35.639989 (+    38us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:35.639996 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.640010 (+    14us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:36.548065 (+908055us) raft_consensus.cc:1685] finished
0812 08:02:36.548067 (+     2us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.548167 (+   100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:35.640996 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:35.641126 (+   130us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:35.641131 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:35.641133 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:35.641442 (+   309us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:35.641453 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:35.641456 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:35.641913 (+   457us) tablet.cc:701] Row locks acquired
0812 08:02:35.641914 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":28416,"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":1378,"prepare.run_cpu_time_us":982,"prepare.run_wall_time_us":980,"thread_start_us":73,"threads_started":1}]]}
I20260812 08:02:36.549232 32084 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.549345 32087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 406) took 273 ms (client timeout 50 ms). Trace:
W20260812 08:02:36.549397 32087 rpcz_store.cc:269] 0812 08:02:36.276259 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:36.276387 (+   128us) service_pool.cc:224] Handling call
0812 08:02:36.548320 (+271933us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.548598 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.548602 (+     4us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.548621 (+    19us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.548626 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.548639 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.549269 (+   630us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:36.549325 (+    56us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624648064}
I20260812 08:02:36.549513 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.549583 32084 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 404) took 859 ms (client timeout 50 ms). Trace:
W20260812 08:02:36.549656 32084 rpcz_store.cc:269] 0812 08:02:35.690470 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.690691 (+   221us) service_pool.cc:224] Handling call
0812 08:02:36.548750 (+858059us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.549392 (+   642us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.549397 (+     5us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.549408 (+    11us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.549413 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.549424 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.549547 (+   123us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:36.549570 (+    23us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1972886784}
W20260812 08:02:36.549820 32089 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:41819 (request call id 405) took 324 ms (client timeout 50 ms). Trace:
W20260812 08:02:36.549892 32089 rpcz_store.cc:269] 0812 08:02:36.225076 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:36.225156 (+    80us) service_pool.cc:224] Handling call
0812 08:02:36.549487 (+324331us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.549639 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.549642 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.549649 (+     7us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.549655 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.549668 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.549774 (+   106us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x555f4e070550. stack: 00007f87dbfe6d35 00007f87dbfe7325 00007f87dbaf382d 0000555f22304b18 0000555f22304b6b 0000555f223089f2 00007f87de037c8a 00007f87df12c38c 00007f87dd790599 00007f87dd794bc4 00007f87dd015f8f 00007f87dd015843 00007f87dd018541 00007f87dd016e9e 00007f87dd0188cb 0000555f2239444d
0812 08:02:36.549805 (+    31us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":745339136}
W20260812 08:02:36.620226 31941 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 435) took 980 ms (client timeout 50 ms). Trace:
I20260812 08:02:36.620297 31936 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.620349 31941 rpcz_store.cc:269] 0812 08:02:35.640368 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.640636 (+   268us) service_pool.cc:224] Handling call
0812 08:02:35.640767 (+   131us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:35.640843 (+    76us) raft_consensus.cc:1534] Early marking committed up to index 0
0812 08:02:35.640845 (+     2us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0812 08:02:35.640959 (+   114us) write_op.cc:270] Start()
0812 08:02:35.640980 (+    21us) write_op.cc:276] Timestamp: P: 1786521755638074 usec, L: 0
0812 08:02:35.641283 (+   303us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:35.641319 (+    36us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:35.641325 (+     6us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:35.641336 (+    11us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0812 08:02:36.620059 (+978723us) raft_consensus.cc:1685] finished
0812 08:02:36.620062 (+     3us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.620191 (+   129us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:35.641537 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:35.641642 (+   105us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:35.641645 (+     3us) write_op.cc:435] Acquired schema lock
0812 08:02:35.641647 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:35.641952 (+   305us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:35.641961 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:35.641964 (+     3us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:35.642162 (+   198us) tablet.cc:701] Row locks acquired
0812 08:02:35.642163 (+     1us) write_op.cc:260] PREPARE: finished
Metrics: {"child_traces":[["op",{"num_ops":50,"partition_lock_wait_us":5,"prepare.queue_time_us":479,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":690,"thread_start_us":104,"threads_started":1}]]}
I20260812 08:02:36.620631 31939 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.620731 31936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 436) took 930 ms (client timeout 50 ms). Trace:
I20260812 08:02:36.620874 31937 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.620800 31936 rpcz_store.cc:269] 0812 08:02:35.690470 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:35.690679 (+   209us) service_pool.cc:224] Handling call
0812 08:02:36.620250 (+929571us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.620489 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.620492 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.620512 (+    20us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.620515 (+     3us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.620526 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.620679 (+   153us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:36.620716 (+    37us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2137236352}
W20260812 08:02:36.620921 31939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 437) took 467 ms (client timeout 50 ms). Trace:
W20260812 08:02:36.621186 31939 rpcz_store.cc:269] 0812 08:02:36.153119 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:36.153268 (+   149us) service_pool.cc:224] Handling call
0812 08:02:36.620600 (+467332us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.620731 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.620734 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.620743 (+     9us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.620750 (+     7us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.620761 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.620882 (+   121us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:36.620909 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1074370304}
W20260812 08:02:36.621284 31937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 438) took 417 ms (client timeout 50 ms). Trace:
I20260812 08:02:36.621300 31942 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.621554 31937 rpcz_store.cc:269] 0812 08:02:36.203990 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:36.204142 (+   152us) service_pool.cc:224] Handling call
0812 08:02:36.620838 (+416696us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.621042 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.621045 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.621066 (+    21us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.621070 (+     4us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.621084 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.621243 (+   159us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:36.621270 (+    27us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957774720}
I20260812 08:02:36.621876 31935 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P 37575efb04814de79c4ef62b1a472a69 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
W20260812 08:02:36.621958 31942 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.208.131:34883 (request call id 439) took 82 ms (client timeout 50 ms). Trace:
W20260812 08:02:36.622013 31942 rpcz_store.cc:269] 0812 08:02:36.539680 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:36.539838 (+   158us) service_pool.cc:224] Handling call
0812 08:02:36.621273 (+ 81435us) raft_consensus.cc:1471] Updating replica for 1 ops
0812 08:02:36.621759 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 74
0812 08:02:36.621762 (+     3us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0812 08:02:36.621771 (+     9us) raft_consensus.cc:1645] Marking committed up to 74
0812 08:02:36.621776 (+     5us) raft_consensus.cc:1697] Filling consensus response to leader.
0812 08:02:36.621791 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0812 08:02:36.621915 (+   124us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x55ffc81ce550. stack: 00007f0f6c236d35 00007f0f6c237325 00007f0f6bd4382d 000055ffc6791b18 000055ffc6791b6b 000055ffc67959f2 00007f0f6e287c8a 00007f0f6f37c38c 00007f0f6d9e0599 00007f0f6d9e4bc4 00007f0f6d265f8f 00007f0f6d265843 00007f0f6d268541 00007f0f6d266e9e 00007f0f6d2688cb 000055ffc682144d
0812 08:02:36.621943 (+    28us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186853120}
I20260812 08:02:36.626035   516 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52728 (ReqId={client: 0e9612641f2244e08914a6efd9914f60, seq_no=73, attempt_no=0}) took 23762 ms. Trace:
I20260812 08:02:36.626191   516 rpcz_store.cc:276] 0812 08:02:12.863994 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:02:12.864158 (+   164us) service_pool.cc:224] Handling call
0812 08:02:36.626013 (+23761855us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:02:34.573305 (+     0us) write_op.cc:183] PREPARE: starting on tablet 049a7ef1624e4420952a5c0960ea2492
0812 08:02:34.573422 (+   117us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:02:34.573427 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:02:34.573429 (+     2us) tablet.cc:662] Decoding operations
0812 08:02:34.573767 (+   338us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:02:34.573777 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:02:34.573782 (+     5us) tablet.cc:685] Acquiring locks for 50 operations
0812 08:02:35.638102 (+1064320us) lock_manager.cc:598] Waited 1064068us for lock on <redacted> (key hash 11785448468237383701)
0812 08:02:35.638114 (+    12us) tablet.cc:701] Row locks acquired
0812 08:02:35.638116 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:02:35.638171 (+    55us) write_op.cc:270] Start()
0812 08:02:35.638196 (+    25us) write_op.cc:276] Timestamp: P: 1786521755638074 usec, L: 0
0812 08:02:35.638198 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:02:35.638342 (+   144us) log.cc:844] Serialized 2464 byte log entry
0812 08:02:36.623029 (+984687us) op_driver.cc:464] REPLICATION: finished
0812 08:02:36.623422 (+   393us) write_op.cc:301] APPLY: starting
0812 08:02:36.623465 (+    43us) tablet.cc:1367] starting BulkCheckPresence
0812 08:02:36.623573 (+   108us) tablet.cc:1370] finished BulkCheckPresence
0812 08:02:36.623575 (+     2us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:02:36.625545 (+  1970us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:02:36.625603 (+    58us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0812 08:02:36.625609 (+     6us) write_op.cc:312] APPLY: finished
0812 08:02:36.625773 (+   164us) log.cc:844] Serialized 319 byte log entry
0812 08:02:36.625843 (+    70us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:02:36.625865 (+    22us) write_op.cc:454] Released schema lock
0812 08:02:36.625873 (+     8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":317,"num_ops":50,"prepare.queue_time_us":21708762,"prepare.run_cpu_time_us":1229,"prepare.run_wall_time_us":1065098,"raft.queue_time_us":51,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":1588,"replication_time_us":984772,"row_lock_wait_count":1,"row_lock_wait_us":1064068,"thread_start_us":145,"threads_started":1}]]}
I20260812 08:02:36.712435 32213 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260812 08:02:36.713667 32089 raft_consensus.cc:1217] T 049a7ef1624e4420952a5c0960ea2492 P ddcf1211ef9942bf910d906bbbe02475 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75]   Dedup: 1.75->[]
I20260812 08:02:36.715093 32068 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:02:36.716279 31920 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 3031281b6b6f431fb18db49ad31f7fb9 | 127.22.208.190:36717 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                   Value                                                                                   |      Tags       |         Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                       | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                       | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                      | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                         | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                      | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                      | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                        | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                       | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:36665 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 37575efb04814de79c4ef62b1a472a69 | 127.22.208.129:37473 | HEALTHY | <none>   |       0        |       0
 dba6326e5e2b4938880eac572a3f5290 | 127.22.208.131:36487 | HEALTHY | <none>   |       1        |       0
 ddcf1211ef9942bf910d906bbbe02475 | 127.22.208.130:45345 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
               Flag               |                                                                                 Value                                                                                 |      Tags       |      Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_server_key_size             | 768                                                                                                                                                                   | experimental    | all 3 server(s) checked
 local_ip_for_outbound_sockets    | 127.22.208.129                                                                                                                                                        | experimental    | 127.22.208.129:37473
 local_ip_for_outbound_sockets    | 127.22.208.130                                                                                                                                                        | experimental    | 127.22.208.130:45345
 local_ip_for_outbound_sockets    | 127.22.208.131                                                                                                                                                        | experimental    | 127.22.208.131:36487
 log_inject_latency               | true                                                                                                                                                                  | runtime,unsafe  | all 3 server(s) checked
 log_inject_latency_ms_mean       | 1000                                                                                                                                                                  | runtime,unsafe  | all 3 server(s) checked
 never_fsync                      | true                                                                                                                                                                  | unsafe,advanced | all 3 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                     | unsafe,hidden   | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                  | unsafe          | all 3 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                    | hidden          | all 3 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden          | 127.22.208.129:37473
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden          | 127.22.208.130:45345
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestHammerOneRow.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden          | 127.22.208.131:36487

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:36665 | all 3 server(s) checked
 time_source         | builtin              | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
   Name    | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 08:02:36.751812 23362 log_verifier.cc:126] Checking tablet 049a7ef1624e4420952a5c0960ea2492
I20260812 08:02:36.817619 23362 log_verifier.cc:177] Verified matching terms for 75 ops in tablet 049a7ef1624e4420952a5c0960ea2492
I20260812 08:02:36.818900 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31849
I20260812 08:02:36.833832 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31997
I20260812 08:02:36.849354 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 32146
I20260812 08:02:36.859881 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 31744
2026-08-12T08:02:36Z chronyd exiting
[       OK ] RaftConsensusITest.TestHammerOneRow (85995 ms)
[ RUN      ] RaftConsensusITest.TestElectionDurationMetrics
I20260812 08:02:36.901988 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:02:36.902074 23362 ts_itest-base.cc:116] --------------
I20260812 08:02:36.902093 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:02:36.902109 23362 ts_itest-base.cc:118] 3 replicas per TS
I20260812 08:02:36.902123 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:02:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:36Z Disabled control of system clock
I20260812 08:02:36.921370 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:34207
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:34207 with env {}
W20260812 08:02:37.067370   558 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:37.067654   558 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:37.067713   558 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:37.073194   558 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:37.073413   558 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:37.073446   558 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:37.073467   558 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:37.078320   558 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:34207
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:34207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:37.079658   558 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:37.081276   558 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:02:37.089440   558 server_base.cc:1061] running on GCE node
W20260812 08:02:37.089327   565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.089366   567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.089368   564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.090365   558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:37.091451   558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:37.092675   558 hybrid_clock.cc:648] HybridClock initialized: now 1786521757092657 us; error 46 us; skew 500 ppm
I20260812 08:02:37.095201   558 webserver.cc:533] Webserver started at http://127.22.208.190:40517/ using document root <none> and password file <none>
I20260812 08:02:37.096040   558 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:37.096155   558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:37.096424   558 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:37.099126   558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0206d04f2e05426986cf3de21c68ea17"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.099800   558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0206d04f2e05426986cf3de21c68ea17"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.103919   558 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:02:37.107057   574 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.108559   558 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:02:37.108896   558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0206d04f2e05426986cf3de21c68ea17"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.109063   558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:37.138425   558 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:37.139266   558 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:37.139493   558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:37.151544   558 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:34207
I20260812 08:02:37.151922   633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:34207 every 8 connection(s)
I20260812 08:02:37.152766   558 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:02:37.153718 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 558
I20260812 08:02:37.153980 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:02:37.157099   634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:37.163053   634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17: Bootstrap starting.
I20260812 08:02:37.165611   634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:37.167023   634 log.cc:826] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:37.169833   634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17: No bootstrap required, opened a new log
I20260812 08:02:37.172740   634 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } }
I20260812 08:02:37.172984   634 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:37.173075   634 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0206d04f2e05426986cf3de21c68ea17, State: Initialized, Role: FOLLOWER
I20260812 08:02:37.173520   634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } }
I20260812 08:02:37.173676   634 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:02:37.173766   634 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:02:37.173887   634 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:37.174943   634 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } }
I20260812 08:02:37.175284   634 leader_election.cc:304] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0206d04f2e05426986cf3de21c68ea17; no voters: 
I20260812 08:02:37.175578   634 leader_election.cc:290] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:02:37.175906   639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:37.176213   639 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [term 1 LEADER]: Becoming Leader. State: Replica: 0206d04f2e05426986cf3de21c68ea17, State: Running, Role: LEADER
I20260812 08:02:37.176532   639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } }
I20260812 08:02:37.176921   634 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:37.179831   641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0206d04f2e05426986cf3de21c68ea17. Latest consensus state: current_term: 1 leader_uuid: "0206d04f2e05426986cf3de21c68ea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } } }
I20260812 08:02:37.180014   641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:37.180310   640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0206d04f2e05426986cf3de21c68ea17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0206d04f2e05426986cf3de21c68ea17" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 34207 } } }
I20260812 08:02:37.180418   640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:37.181483   653 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:37.184967   653 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:37.192862   653 catalog_manager.cc:1383] Generated new cluster ID: 994da6a2033540b5b490777f414964ac
I20260812 08:02:37.192970   653 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:37.221045   653 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:37.222131   653 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:37.232884   653 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 0206d04f2e05426986cf3de21c68ea17: Generated new TSK 0
I20260812 08:02:37.233630   653 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:37.247151 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--builtin_ntp_servers=127.22.208.148:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100 with env {}
W20260812 08:02:37.399617   660 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:37.399923   660 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:37.399976   660 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:37.404984   660 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:37.405139   660 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:02:37.410013   660 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--never_fsync=true
--heap_profile_path=/tmp/kudu.660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:37.411353   660 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:37.412590   660 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:37.415691   660 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:02:37.420132   665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.420419   660 server_base.cc:1061] running on GCE node
W20260812 08:02:37.420439   668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.420164   666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.421016   660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:37.421646   660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:37.422816   660 hybrid_clock.cc:648] HybridClock initialized: now 1786521757422790 us; error 40 us; skew 500 ppm
I20260812 08:02:37.425091   660 webserver.cc:533] Webserver started at http://127.22.208.129:39735/ using document root <none> and password file <none>
I20260812 08:02:37.425756   660 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:37.425822   660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:37.426055   660 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:37.427906   660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "273a88b5689047e7b6a6f80b32c9fb19"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.428508   660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "273a88b5689047e7b6a6f80b32c9fb19"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.432502   660 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:37.435369   675 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.436695   660 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:37.436853   660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "273a88b5689047e7b6a6f80b32c9fb19"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.436985   660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:37.464803   660 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:37.465624   660 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:37.465920   660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:37.466660   660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:37.467779   660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:37.467829   660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.467871   660 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:37.467887   660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.481405   660 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:46817
I20260812 08:02:37.481505   802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:46817 every 8 connection(s)
I20260812 08:02:37.482630   660 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:02:37.488359 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 660
I20260812 08:02:37.488536 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:02:37.491511 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--builtin_ntp_servers=127.22.208.148:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100 with env {}
I20260812 08:02:37.493088   803 heartbeater.cc:344] Connected to a master server at 127.22.208.190:34207
I20260812 08:02:37.493369   803 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:37.494042   803 heartbeater.cc:507] Master 127.22.208.190:34207 requested a full tablet report, sending...
I20260812 08:02:37.496101   593 ts_manager.cc:194] Registered new tserver with Master: 273a88b5689047e7b6a6f80b32c9fb19 (127.22.208.129:46817)
I20260812 08:02:37.497532   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:49027
W20260812 08:02:37.619855   806 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:37.620150   806 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:37.620260   806 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:37.624545   806 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:37.624764   806 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:02:37.629647   806 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--never_fsync=true
--heap_profile_path=/tmp/kudu.806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:37.631286   806 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:37.632563   806 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:37.635855   806 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:02:37.640916   816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.640928   811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.641237   806 server_base.cc:1061] running on GCE node
W20260812 08:02:37.641549   812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.642171   806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:37.643043   806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:37.644335   806 hybrid_clock.cc:648] HybridClock initialized: now 1786521757644319 us; error 41 us; skew 500 ppm
I20260812 08:02:37.646988   806 webserver.cc:533] Webserver started at http://127.22.208.130:44213/ using document root <none> and password file <none>
I20260812 08:02:37.647670   806 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:37.647768   806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:37.648010   806 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:37.649706   806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5ddb2b49f845468d9e4c12969cd17844"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.650249   806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5ddb2b49f845468d9e4c12969cd17844"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.654349   806 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260812 08:02:37.657665   822 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.659798   806 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:02:37.659943   806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5ddb2b49f845468d9e4c12969cd17844"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.660106   806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:37.680610   806 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:37.681670   806 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:37.681865   806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:37.682950   806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:37.684119   806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:37.684176   806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.684221   806 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:37.684237   806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.697008   806 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:44409
I20260812 08:02:37.697036   952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:44409 every 8 connection(s)
I20260812 08:02:37.698098   806 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:02:37.701655 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 806
I20260812 08:02:37.701807 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:02:37.704586 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--builtin_ntp_servers=127.22.208.148:45847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100 with env {}
I20260812 08:02:37.709942   953 heartbeater.cc:344] Connected to a master server at 127.22.208.190:34207
I20260812 08:02:37.710511   953 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:37.711344   953 heartbeater.cc:507] Master 127.22.208.190:34207 requested a full tablet report, sending...
I20260812 08:02:37.712922   593 ts_manager.cc:194] Registered new tserver with Master: 5ddb2b49f845468d9e4c12969cd17844 (127.22.208.130:44409)
I20260812 08:02:37.713589   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:42339
W20260812 08:02:37.845111   956 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:37.845409   956 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:37.845486   956 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:37.849454   956 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:37.849741   956 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:02:37.854363   956 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:45847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:34207
--never_fsync=true
--heap_profile_path=/tmp/kudu.956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:37.856127   956 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:37.857445   956 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:02:37.860608   956 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:02:37.866956   961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.866925   964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:37.867211   962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:37.867007   956 server_base.cc:1061] running on GCE node
I20260812 08:02:37.867784   956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:37.868413   956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:37.869654   956 hybrid_clock.cc:648] HybridClock initialized: now 1786521757869630 us; error 38 us; skew 500 ppm
I20260812 08:02:37.871982   956 webserver.cc:533] Webserver started at http://127.22.208.131:37737/ using document root <none> and password file <none>
I20260812 08:02:37.872946   956 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:37.873045   956 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:37.873389   956 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:37.875989   956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "909f632aeb3140b79e16578f1d92a5f7"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.876609   956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "909f632aeb3140b79e16578f1d92a5f7"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.881470   956 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20260812 08:02:37.884999   970 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.886430   956 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260812 08:02:37.886732   956 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "909f632aeb3140b79e16578f1d92a5f7"
format_stamp: "Formatted at 2026-08-12 08:02:37 on dist-test-slave-dhph"
I20260812 08:02:37.886868   956 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:37.922044   956 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:37.922933   956 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:37.923161   956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:37.923918   956 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:37.925079   956 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:37.925168   956 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.925239   956 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:37.925290   956 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:37.940534   956 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:36793
I20260812 08:02:37.940555  1096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:36793 every 8 connection(s)
I20260812 08:02:37.942322   956 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:02:37.945978 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 956
I20260812 08:02:37.946187 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusITest.TestElectionDurationMetrics.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:02:37.954036  1097 heartbeater.cc:344] Connected to a master server at 127.22.208.190:34207
I20260812 08:02:37.954572  1097 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:37.955346  1097 heartbeater.cc:507] Master 127.22.208.190:34207 requested a full tablet report, sending...
I20260812 08:02:37.956912   593 ts_manager.cc:194] Registered new tserver with Master: 909f632aeb3140b79e16578f1d92a5f7 (127.22.208.131:36793)
I20260812 08:02:37.957701   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:58089
I20260812 08:02:37.962848 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:37.976675   593 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43846:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 08:02:37.980156   593 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:02:38.005656   877 tablet_service.cc:1511] Processing CreateTablet for tablet fdd90b8eeb8841969b53f564e0ee1e5e (DEFAULT_TABLE table=TestTable [id=c574febf89d54e249507e02a890b1d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:38.005656  1024 tablet_service.cc:1511] Processing CreateTablet for tablet fdd90b8eeb8841969b53f564e0ee1e5e (DEFAULT_TABLE table=TestTable [id=c574febf89d54e249507e02a890b1d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:38.007166  1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd90b8eeb8841969b53f564e0ee1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:38.007167   877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd90b8eeb8841969b53f564e0ee1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:38.006930   730 tablet_service.cc:1511] Processing CreateTablet for tablet fdd90b8eeb8841969b53f564e0ee1e5e (DEFAULT_TABLE table=TestTable [id=c574febf89d54e249507e02a890b1d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:38.008787   730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd90b8eeb8841969b53f564e0ee1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:38.017449  1116 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: Bootstrap starting.
I20260812 08:02:38.017812  1117 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Bootstrap starting.
I20260812 08:02:38.020238  1118 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: Bootstrap starting.
I20260812 08:02:38.020857  1116 tablet_bootstrap.cc:654] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:38.022935  1116 log.cc:826] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:38.023398  1117 tablet_bootstrap.cc:654] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:38.024376  1117 log.cc:826] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:38.024581  1118 tablet_bootstrap.cc:654] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:38.025451  1118 log.cc:826] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:38.026263  1116 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: No bootstrap required, opened a new log
I20260812 08:02:38.026558  1116 ts_tablet_manager.cc:1403] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20260812 08:02:38.026672  1117 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: No bootstrap required, opened a new log
I20260812 08:02:38.026875  1117 ts_tablet_manager.cc:1403] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Time spent bootstrapping tablet: real 0.009s	user 0.009s	sys 0.000s
I20260812 08:02:38.027576  1118 tablet_bootstrap.cc:492] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: No bootstrap required, opened a new log
I20260812 08:02:38.027915  1118 ts_tablet_manager.cc:1403] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20260812 08:02:38.030369  1117 raft_consensus.cc:359] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.030346  1116 raft_consensus.cc:359] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.030625  1117 raft_consensus.cc:385] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:38.030643  1116 raft_consensus.cc:385] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:38.030695  1117 raft_consensus.cc:740] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909f632aeb3140b79e16578f1d92a5f7, State: Initialized, Role: FOLLOWER
I20260812 08:02:38.030706  1116 raft_consensus.cc:740] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddb2b49f845468d9e4c12969cd17844, State: Initialized, Role: FOLLOWER
I20260812 08:02:38.031167  1116 consensus_queue.cc:260] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.031106  1118 raft_consensus.cc:359] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.031239  1117 consensus_queue.cc:260] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.031325  1118 raft_consensus.cc:385] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:38.031375  1118 raft_consensus.cc:740] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273a88b5689047e7b6a6f80b32c9fb19, State: Initialized, Role: FOLLOWER
I20260812 08:02:38.031852  1118 consensus_queue.cc:260] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.031948   953 heartbeater.cc:499] Master 127.22.208.190:34207 was elected leader, sending a full tablet report...
I20260812 08:02:38.032195  1116 ts_tablet_manager.cc:1434] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:02:38.032208  1117 ts_tablet_manager.cc:1434] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:02:38.032524  1097 heartbeater.cc:499] Master 127.22.208.190:34207 was elected leader, sending a full tablet report...
I20260812 08:02:38.032619  1118 ts_tablet_manager.cc:1434] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:02:38.033223   803 heartbeater.cc:499] Master 127.22.208.190:34207 was elected leader, sending a full tablet report...
I20260812 08:02:38.052626  1122 raft_consensus.cc:493] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:38.053352  1122 raft_consensus.cc:515] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.054703  1122 leader_election.cc:290] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909f632aeb3140b79e16578f1d92a5f7 (127.22.208.131:36793), 273a88b5689047e7b6a6f80b32c9fb19 (127.22.208.129:46817)
I20260812 08:02:38.061061  1046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd90b8eeb8841969b53f564e0ee1e5e" candidate_uuid: "5ddb2b49f845468d9e4c12969cd17844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909f632aeb3140b79e16578f1d92a5f7" is_pre_election: true
I20260812 08:02:38.061497   753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd90b8eeb8841969b53f564e0ee1e5e" candidate_uuid: "5ddb2b49f845468d9e4c12969cd17844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273a88b5689047e7b6a6f80b32c9fb19" is_pre_election: true
I20260812 08:02:38.061563  1046 raft_consensus.cc:2468] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ddb2b49f845468d9e4c12969cd17844 in term 0.
I20260812 08:02:38.061755   753 raft_consensus.cc:2468] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ddb2b49f845468d9e4c12969cd17844 in term 0.
I20260812 08:02:38.062613   825 leader_election.cc:304] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273a88b5689047e7b6a6f80b32c9fb19, 5ddb2b49f845468d9e4c12969cd17844; no voters: 
I20260812 08:02:38.062927  1122 raft_consensus.cc:2804] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:38.063027  1122 raft_consensus.cc:493] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:38.063084  1122 raft_consensus.cc:3060] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:38.064428  1122 raft_consensus.cc:515] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.064890  1122 leader_election.cc:290] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [CANDIDATE]: Term 1 election: Requested vote from peers 909f632aeb3140b79e16578f1d92a5f7 (127.22.208.131:36793), 273a88b5689047e7b6a6f80b32c9fb19 (127.22.208.129:46817)
I20260812 08:02:38.065778   753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd90b8eeb8841969b53f564e0ee1e5e" candidate_uuid: "5ddb2b49f845468d9e4c12969cd17844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273a88b5689047e7b6a6f80b32c9fb19"
I20260812 08:02:38.065972   753 raft_consensus.cc:3060] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:38.066253  1046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd90b8eeb8841969b53f564e0ee1e5e" candidate_uuid: "5ddb2b49f845468d9e4c12969cd17844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909f632aeb3140b79e16578f1d92a5f7"
I20260812 08:02:38.066414  1046 raft_consensus.cc:3060] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:38.067570   753 raft_consensus.cc:2468] T fdd90b8eeb8841969b53f564e0ee1e5e P 273a88b5689047e7b6a6f80b32c9fb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ddb2b49f845468d9e4c12969cd17844 in term 1.
I20260812 08:02:38.067772  1046 raft_consensus.cc:2468] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ddb2b49f845468d9e4c12969cd17844 in term 1.
I20260812 08:02:38.068100   825 leader_election.cc:304] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273a88b5689047e7b6a6f80b32c9fb19, 5ddb2b49f845468d9e4c12969cd17844; no voters: 
I20260812 08:02:38.068384  1122 raft_consensus.cc:2804] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:38.068712  1122 raft_consensus.cc:697] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [term 1 LEADER]: Becoming Leader. State: Replica: 5ddb2b49f845468d9e4c12969cd17844, State: Running, Role: LEADER
I20260812 08:02:38.069096  1122 consensus_queue.cc:237] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } }
I20260812 08:02:38.075330   592 catalog_manager.cc:5719] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ddb2b49f845468d9e4c12969cd17844 (127.22.208.130). New cstate: current_term: 1 leader_uuid: "5ddb2b49f845468d9e4c12969cd17844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddb2b49f845468d9e4c12969cd17844" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:38.089475 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:38.091961 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 273a88b5689047e7b6a6f80b32c9fb19 to finish bootstrapping
I20260812 08:02:38.100481 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5ddb2b49f845468d9e4c12969cd17844 to finish bootstrapping
I20260812 08:02:38.108886 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 909f632aeb3140b79e16578f1d92a5f7 to finish bootstrapping
I20260812 08:02:38.136039 23362 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0 0.0
I20260812 08:02:38.189195  1122 consensus_queue.cc:1048] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909f632aeb3140b79e16578f1d92a5f7" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 36793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 08:02:38.193550  1098 tablet.cc:2406] T fdd90b8eeb8841969b53f564e0ee1e5e P 909f632aeb3140b79e16578f1d92a5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:38.195228  1126 consensus_queue.cc:1048] T fdd90b8eeb8841969b53f564e0ee1e5e P 5ddb2b49f845468d9e4c12969cd17844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273a88b5689047e7b6a6f80b32c9fb19" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:38.254086 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 660
I20260812 08:02:38.266027 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 806
I20260812 08:02:38.277768 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 956
I20260812 08:02:38.287922 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 558
2026-08-12T08:02:38Z chronyd exiting
[       OK ] RaftConsensusITest.TestElectionDurationMetrics (1415 ms)
[----------] 6 tests from RaftConsensusITest (132469 ms total)

[----------] 1 test from EncryptionEnabled/RaftConsensusParamEncryptionITest
[ RUN      ] EncryptionEnabled/RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted/1
I20260812 08:02:38.316501 23362 ts_itest-base.cc:115] Starting cluster with:
I20260812 08:02:38.316582 23362 ts_itest-base.cc:116] --------------
I20260812 08:02:38.316601 23362 ts_itest-base.cc:117] 3 tablet servers
I20260812 08:02:38.316617 23362 ts_itest-base.cc:118] 3 replicas per TS
I20260812 08:02:38.316644 23362 ts_itest-base.cc:119] --------------
2026-08-12T08:02:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:38Z Disabled control of system clock
I20260812 08:02:38.328842 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:33325
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:33325
--encrypt_data_at_rest=true with env {}
W20260812 08:02:38.464835  1145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:38.465114  1145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:38.465171  1145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:38.468983  1145 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:38.469069  1145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:38.469092  1145 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:38.469112  1145 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:38.473469  1145 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:33325
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:38.474900  1145 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:38.476300  1145 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:38.483310  1154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:38.483517  1145 server_base.cc:1061] running on GCE node
W20260812 08:02:38.483319  1151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:38.483301  1152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:38.484424  1145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:38.485819  1145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:38.487042  1145 hybrid_clock.cc:648] HybridClock initialized: now 1786521758487012 us; error 46 us; skew 500 ppm
I20260812 08:02:38.489672  1145 webserver.cc:533] Webserver started at http://127.22.208.190:35925/ using document root <none> and password file <none>
I20260812 08:02:38.490731  1145 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:38.490882  1145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:38.491129  1145 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:38.493440  1145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2e37a56783024e73beffff68156ee0b0"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "990730fd5146e2cac355bc03fb35f2f2"
server_key_iv: "a503bc77669ee216dde4c683567f2e92"
server_key_version: "encryptionkey@0"
I20260812 08:02:38.494016  1145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2e37a56783024e73beffff68156ee0b0"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "990730fd5146e2cac355bc03fb35f2f2"
server_key_iv: "a503bc77669ee216dde4c683567f2e92"
server_key_version: "encryptionkey@0"
I20260812 08:02:38.497805  1145 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 08:02:38.500499  1162 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:38.501822  1145 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260812 08:02:38.501991  1145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2e37a56783024e73beffff68156ee0b0"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "990730fd5146e2cac355bc03fb35f2f2"
server_key_iv: "a503bc77669ee216dde4c683567f2e92"
server_key_version: "encryptionkey@0"
I20260812 08:02:38.502123  1145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:38.540055  1145 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:38.540843  1145 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:38.541086  1145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:38.550064  1145 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:33325
I20260812 08:02:38.550112  1222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:33325 every 8 connection(s)
I20260812 08:02:38.551250  1145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260812 08:02:38.554826  1223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:38.558964 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1145
I20260812 08:02:38.559145 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260812 08:02:38.559572 23362 external_mini_cluster.cc:1474] Setting key b32d1ad77b6cc8e0e97f9629d11fd8d8
I20260812 08:02:38.561779  1223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0: Bootstrap starting.
I20260812 08:02:38.564682  1223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:38.565456  1223 log.cc:826] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:38.567883  1223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0: No bootstrap required, opened a new log
I20260812 08:02:38.571468  1223 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } }
I20260812 08:02:38.571710  1223 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:38.571749  1223 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e37a56783024e73beffff68156ee0b0, State: Initialized, Role: FOLLOWER
I20260812 08:02:38.572160  1223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } }
I20260812 08:02:38.572324  1223 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:02:38.572429  1223 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:02:38.572567  1223 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:38.573822  1223 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } }
I20260812 08:02:38.574817  1223 leader_election.cc:304] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e37a56783024e73beffff68156ee0b0; no voters: 
I20260812 08:02:38.575237  1223 leader_election.cc:290] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:02:38.575531  1229 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:38.575851  1229 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e37a56783024e73beffff68156ee0b0, State: Running, Role: LEADER
I20260812 08:02:38.576218  1229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } }
I20260812 08:02:38.576789  1223 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:38.578115  1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e37a56783024e73beffff68156ee0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } } }
I20260812 08:02:38.578296  1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:38.578115  1231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e37a56783024e73beffff68156ee0b0. Latest consensus state: current_term: 1 leader_uuid: "2e37a56783024e73beffff68156ee0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37a56783024e73beffff68156ee0b0" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 33325 } } }
I20260812 08:02:38.578853  1231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:38.580171  1238 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:38.584038  1238 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:38.590440  1238 catalog_manager.cc:1383] Generated new cluster ID: 417c42a08dd84964a5a4511a1c83fde7
I20260812 08:02:38.590540  1238 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:38.620406  1238 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:38.621649  1238 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:38.634769  1238 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 2e37a56783024e73beffff68156ee0b0: Generated new TSK 0
I20260812 08:02:38.635567  1238 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:38.643628 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:33325
--builtin_ntp_servers=127.22.208.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--consensus_max_batch_size_bytes=500000
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--maintenance_manager_polling_interval_ms=100
--flush_threshold_secs=3
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--encrypt_data_at_rest=true
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0 with env {}
W20260812 08:02:38.786065  1248 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 08:02:38.786384  1248 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:38.786408  1248 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:38.786550  1248 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:38.790442  1248 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 08:02:38.790546  1248 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:38.790696  1248 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=3
W20260812 08:02:38.790771  1248 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:02:38.795562  1248 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--consensus_max_batch_size_bytes=500000
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0
--webserver_interface=127.22.208.129
--webserver_port=0
--flush_threshold_secs=3
--tserver_master_addrs=127.22.208.190:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:38.797245  1248 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:38.798480  1248 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:38.806135  1254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:38.806249  1253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:38.806550  1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:38.806924  1248 server_base.cc:1061] running on GCE node
I20260812 08:02:38.807616  1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:38.808490  1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:38.809705  1248 hybrid_clock.cc:648] HybridClock initialized: now 1786521758809681 us; error 41 us; skew 500 ppm
I20260812 08:02:38.812280  1248 webserver.cc:533] Webserver started at http://127.22.208.129:44335/ using document root <none> and password file <none>
I20260812 08:02:38.813226  1248 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:38.813305  1248 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:38.813488  1248 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:38.815317  1248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e719176bf8b84aabbe7573da7d3a9d80"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:38.815779  1248 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e719176bf8b84aabbe7573da7d3a9d80"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:38.819754  1248 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260812 08:02:38.822428  1263 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:38.823961  1248 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:02:38.824097  1248 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e719176bf8b84aabbe7573da7d3a9d80"
format_stamp: "Formatted at 2026-08-12 08:02:38 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:38.824424  1248 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:38.846937  1248 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:38.847939  1248 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:38.848348  1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:38.849092  1248 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:38.850474  1248 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:38.850754  1248 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20260812 08:02:38.850852  1248 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:38.850873  1248 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:38.863865  1248 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:38987
I20260812 08:02:38.863924  1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:38987 every 8 connection(s)
I20260812 08:02:38.865254  1248 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260812 08:02:38.874085 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1248
I20260812 08:02:38.874325 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260812 08:02:38.874737 23362 external_mini_cluster.cc:1474] Setting key 2a2b28292e2f2c2d2223202126272400
I20260812 08:02:38.878031 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:33325
--builtin_ntp_servers=127.22.208.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--consensus_max_batch_size_bytes=500000
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--maintenance_manager_polling_interval_ms=100
--flush_threshold_secs=3
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--encrypt_data_at_rest=true
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0 with env {}
I20260812 08:02:38.881031  1395 heartbeater.cc:344] Connected to a master server at 127.22.208.190:33325
I20260812 08:02:38.881457  1395 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:38.882121  1395 heartbeater.cc:507] Master 127.22.208.190:33325 requested a full tablet report, sending...
I20260812 08:02:38.884459  1182 ts_manager.cc:194] Registered new tserver with Master: e719176bf8b84aabbe7573da7d3a9d80 (127.22.208.129:38987)
I20260812 08:02:38.886534  1182 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:44897
W20260812 08:02:39.034634  1398 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 08:02:39.034960  1398 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:39.034983  1398 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:39.035043  1398 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:39.039021  1398 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 08:02:39.039125  1398 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:39.039178  1398 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=3
W20260812 08:02:39.039235  1398 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:02:39.044044  1398 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--consensus_max_batch_size_bytes=500000
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0
--webserver_interface=127.22.208.130
--webserver_port=0
--flush_threshold_secs=3
--tserver_master_addrs=127.22.208.190:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:39.045287  1398 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:39.046519  1398 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:39.056008  1403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:39.056069  1404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:39.056067  1406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:39.056905  1398 server_base.cc:1061] running on GCE node
I20260812 08:02:39.057389  1398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:39.057991  1398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:39.059325  1398 hybrid_clock.cc:648] HybridClock initialized: now 1786521759059230 us; error 112 us; skew 500 ppm
I20260812 08:02:39.061702  1398 webserver.cc:533] Webserver started at http://127.22.208.130:41433/ using document root <none> and password file <none>
I20260812 08:02:39.062358  1398 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:39.062456  1398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:39.062785  1398 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:39.064802  1398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8bcce3942e2e480191b4f6809e1377fd"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.065413  1398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8bcce3942e2e480191b4f6809e1377fd"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.069931  1398 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260812 08:02:39.073433  1412 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.075207  1398 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:02:39.075403  1398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8bcce3942e2e480191b4f6809e1377fd"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.075546  1398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:39.100483  1398 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:39.101267  1398 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:39.101488  1398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:39.102139  1398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:39.103466  1398 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:39.103559  1398 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.103684  1398 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:39.103896  1398 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.121001  1398 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:41643
I20260812 08:02:39.121085  1542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:41643 every 8 connection(s)
I20260812 08:02:39.122154  1398 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260812 08:02:39.130800 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1398
I20260812 08:02:39.131125 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260812 08:02:39.131484 23362 external_mini_cluster.cc:1474] Setting key 2a2b28292e2f2c2d2223202126272400
I20260812 08:02:39.134133 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:33325
--builtin_ntp_servers=127.22.208.148:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=1
--consensus_max_batch_size_bytes=500000
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--maintenance_manager_polling_interval_ms=100
--flush_threshold_secs=3
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--encrypt_data_at_rest=true
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0 with env {}
I20260812 08:02:39.135576  1543 heartbeater.cc:344] Connected to a master server at 127.22.208.190:33325
I20260812 08:02:39.135986  1543 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:39.136652  1543 heartbeater.cc:507] Master 127.22.208.190:33325 requested a full tablet report, sending...
I20260812 08:02:39.138280  1182 ts_manager.cc:194] Registered new tserver with Master: 8bcce3942e2e480191b4f6809e1377fd (127.22.208.130:41643)
I20260812 08:02:39.139186  1182 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:35011
W20260812 08:02:39.268612  1547 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260812 08:02:39.268896  1547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:39.268919  1547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:39.268975  1547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:39.272643  1547 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260812 08:02:39.272742  1547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:39.272792  1547 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=3
W20260812 08:02:39.272892  1547 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:02:39.278092  1547 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--consensus_max_batch_size_bytes=500000
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--test_server_key=000102030405060708090a0b0c0d0e2a
--test_server_key_iv=2a0e0d0c0b0a09080706050403020100
--test_server_key_version=kudutenantkey@0
--webserver_interface=127.22.208.131
--webserver_port=0
--flush_threshold_secs=3
--tserver_master_addrs=127.22.208.190:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:39.280181  1547 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:39.282780  1547 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:39.291064  1553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:39.291216  1552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:39.291783  1547 server_base.cc:1061] running on GCE node
W20260812 08:02:39.292915  1555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:39.293774  1547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:39.294536  1547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:39.295794  1547 hybrid_clock.cc:648] HybridClock initialized: now 1786521759295673 us; error 142 us; skew 500 ppm
I20260812 08:02:39.298264  1547 webserver.cc:533] Webserver started at http://127.22.208.131:43487/ using document root <none> and password file <none>
I20260812 08:02:39.298995  1547 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:39.299089  1547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:39.299320  1547 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:39.301046  1547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "07e86a1b785c452a9a2f28d5bd3a70f5"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.301669  1547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "07e86a1b785c452a9a2f28d5bd3a70f5"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.305881  1547 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.001s
I20260812 08:02:39.311539  1562 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.313128  1547 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
I20260812 08:02:39.313318  1547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "07e86a1b785c452a9a2f28d5bd3a70f5"
format_stamp: "Formatted at 2026-08-12 08:02:39 on dist-test-slave-dhph"
server_key: "000102030405060708090a0b0c0d0e2a"
server_key_iv: "2a0e0d0c0b0a09080706050403020100"
server_key_version: "kudutenantkey@0"
I20260812 08:02:39.313472  1547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:39.338377  1547 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:39.339666  1547 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:39.339901  1547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:39.340565  1547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:39.341706  1547 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:39.341785  1547 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.341862  1547 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:39.341907  1547 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:39.361724  1547 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:32793
I20260812 08:02:39.361901  1690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:32793 every 8 connection(s)
I20260812 08:02:39.362952  1547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260812 08:02:39.367621 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1547
I20260812 08:02:39.368019 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260812 08:02:39.368369 23362 external_mini_cluster.cc:1474] Setting key 2a2b28292e2f2c2d2223202126272400
I20260812 08:02:39.376394  1692 heartbeater.cc:344] Connected to a master server at 127.22.208.190:33325
I20260812 08:02:39.376852  1692 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:39.377630  1692 heartbeater.cc:507] Master 127.22.208.190:33325 requested a full tablet report, sending...
I20260812 08:02:39.379556  1182 ts_manager.cc:194] Registered new tserver with Master: 07e86a1b785c452a9a2f28d5bd3a70f5 (127.22.208.131:32793)
I20260812 08:02:39.380263  1182 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:51703
I20260812 08:02:39.383684 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:39.399698  1182 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46728:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
W20260812 08:02:39.402269  1182 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:02:39.429388  1613 tablet_service.cc:1511] Processing CreateTablet for tablet b8189424ca9e4aa7b21e9da93562acaf (DEFAULT_TABLE table=TestTable [id=4212f7ffa906452b91957761eadf4046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:39.429994  1321 tablet_service.cc:1511] Processing CreateTablet for tablet b8189424ca9e4aa7b21e9da93562acaf (DEFAULT_TABLE table=TestTable [id=4212f7ffa906452b91957761eadf4046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:39.430297  1466 tablet_service.cc:1511] Processing CreateTablet for tablet b8189424ca9e4aa7b21e9da93562acaf (DEFAULT_TABLE table=TestTable [id=4212f7ffa906452b91957761eadf4046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:02:39.431149  1321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8189424ca9e4aa7b21e9da93562acaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:39.431131  1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8189424ca9e4aa7b21e9da93562acaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:39.431254  1466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8189424ca9e4aa7b21e9da93562acaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:39.439055  1710 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Bootstrap starting.
I20260812 08:02:39.440337  1712 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Bootstrap starting.
I20260812 08:02:39.440482  1711 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Bootstrap starting.
I20260812 08:02:39.442202  1710 tablet_bootstrap.cc:654] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:39.442812  1711 tablet_bootstrap.cc:654] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:39.442812  1712 tablet_bootstrap.cc:654] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:39.443161  1710 log.cc:826] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:39.443632  1712 log.cc:826] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:39.443632  1711 log.cc:826] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:39.445291  1712 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: No bootstrap required, opened a new log
I20260812 08:02:39.445302  1710 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: No bootstrap required, opened a new log
I20260812 08:02:39.445515  1710 ts_tablet_manager.cc:1403] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20260812 08:02:39.445518  1712 ts_tablet_manager.cc:1403] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:02:39.446998  1711 tablet_bootstrap.cc:492] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: No bootstrap required, opened a new log
I20260812 08:02:39.447357  1711 ts_tablet_manager.cc:1403] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20260812 08:02:39.448825  1712 raft_consensus.cc:359] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.449064  1712 raft_consensus.cc:385] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:39.449149  1712 raft_consensus.cc:740] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcce3942e2e480191b4f6809e1377fd, State: Initialized, Role: FOLLOWER
I20260812 08:02:39.449177  1710 raft_consensus.cc:359] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.449383  1710 raft_consensus.cc:385] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:39.449442  1710 raft_consensus.cc:740] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e86a1b785c452a9a2f28d5bd3a70f5, State: Initialized, Role: FOLLOWER
I20260812 08:02:39.449712  1712 consensus_queue.cc:260] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.449867  1710 consensus_queue.cc:260] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.450508  1543 heartbeater.cc:499] Master 127.22.208.190:33325 was elected leader, sending a full tablet report...
I20260812 08:02:39.450825  1712 ts_tablet_manager.cc:1434] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.005s
I20260812 08:02:39.450853  1711 raft_consensus.cc:359] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.451052  1711 raft_consensus.cc:385] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:39.451109  1711 raft_consensus.cc:740] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e719176bf8b84aabbe7573da7d3a9d80, State: Initialized, Role: FOLLOWER
I20260812 08:02:39.451532  1710 ts_tablet_manager.cc:1434] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:02:39.451519  1711 consensus_queue.cc:260] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.451897  1692 heartbeater.cc:499] Master 127.22.208.190:33325 was elected leader, sending a full tablet report...
I20260812 08:02:39.452325  1711 ts_tablet_manager.cc:1434] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:02:39.452755  1395 heartbeater.cc:499] Master 127.22.208.190:33325 was elected leader, sending a full tablet report...
W20260812 08:02:39.464143  1693 tablet.cc:2406] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:02:39.468758  1396 tablet.cc:2406] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:02:39.524472  1544 tablet.cc:2406] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:39.692713  1716 raft_consensus.cc:493] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:39.692989  1716 raft_consensus.cc:515] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.694231  1716 leader_election.cc:290] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07e86a1b785c452a9a2f28d5bd3a70f5 (127.22.208.131:32793), e719176bf8b84aabbe7573da7d3a9d80 (127.22.208.129:38987)
I20260812 08:02:39.700363  1343 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8189424ca9e4aa7b21e9da93562acaf" candidate_uuid: "8bcce3942e2e480191b4f6809e1377fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e719176bf8b84aabbe7573da7d3a9d80" is_pre_election: true
I20260812 08:02:39.700737  1343 raft_consensus.cc:2468] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcce3942e2e480191b4f6809e1377fd in term 0.
I20260812 08:02:39.700682  1635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8189424ca9e4aa7b21e9da93562acaf" candidate_uuid: "8bcce3942e2e480191b4f6809e1377fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" is_pre_election: true
I20260812 08:02:39.701006  1635 raft_consensus.cc:2468] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcce3942e2e480191b4f6809e1377fd in term 0.
I20260812 08:02:39.701393  1413 leader_election.cc:304] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bcce3942e2e480191b4f6809e1377fd, e719176bf8b84aabbe7573da7d3a9d80; no voters: 
I20260812 08:02:39.701746  1716 raft_consensus.cc:2804] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:39.701845  1716 raft_consensus.cc:493] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:39.701933  1716 raft_consensus.cc:3060] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:39.703187  1716 raft_consensus.cc:515] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.703701  1716 leader_election.cc:290] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [CANDIDATE]: Term 1 election: Requested vote from peers 07e86a1b785c452a9a2f28d5bd3a70f5 (127.22.208.131:32793), e719176bf8b84aabbe7573da7d3a9d80 (127.22.208.129:38987)
I20260812 08:02:39.704172  1635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8189424ca9e4aa7b21e9da93562acaf" candidate_uuid: "8bcce3942e2e480191b4f6809e1377fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5"
I20260812 08:02:39.704413  1343 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8189424ca9e4aa7b21e9da93562acaf" candidate_uuid: "8bcce3942e2e480191b4f6809e1377fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e719176bf8b84aabbe7573da7d3a9d80"
I20260812 08:02:39.704345  1635 raft_consensus.cc:3060] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:39.704541  1343 raft_consensus.cc:3060] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:39.705972  1343 raft_consensus.cc:2468] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcce3942e2e480191b4f6809e1377fd in term 1.
I20260812 08:02:39.706007  1635 raft_consensus.cc:2468] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcce3942e2e480191b4f6809e1377fd in term 1.
I20260812 08:02:39.706477  1413 leader_election.cc:304] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bcce3942e2e480191b4f6809e1377fd, e719176bf8b84aabbe7573da7d3a9d80; no voters: 
I20260812 08:02:39.706801  1716 raft_consensus.cc:2804] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:39.707247  1716 raft_consensus.cc:697] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [term 1 LEADER]: Becoming Leader. State: Replica: 8bcce3942e2e480191b4f6809e1377fd, State: Running, Role: LEADER
I20260812 08:02:39.707621  1716 consensus_queue.cc:237] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } }
I20260812 08:02:39.711776  1174 catalog_manager.cc:5719] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bcce3942e2e480191b4f6809e1377fd (127.22.208.130). New cstate: current_term: 1 leader_uuid: "8bcce3942e2e480191b4f6809e1377fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcce3942e2e480191b4f6809e1377fd" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:39.725287 23362 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:02:39.727301 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e719176bf8b84aabbe7573da7d3a9d80 to finish bootstrapping
I20260812 08:02:39.734381 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8bcce3942e2e480191b4f6809e1377fd to finish bootstrapping
I20260812 08:02:39.741564 23362 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 07e86a1b785c452a9a2f28d5bd3a70f5 to finish bootstrapping
I20260812 08:02:39.759392 23362 raft_consensus-itest.cc:851] Paused one of the replicas, starting to write.
I20260812 08:02:39.776211  1635 raft_consensus.cc:1275] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5 [term 1 FOLLOWER]: Refusing update from remote peer 8bcce3942e2e480191b4f6809e1377fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:39.777043  1721 consensus_queue.cc:1048] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "07e86a1b785c452a9a2f28d5bd3a70f5" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 32793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:39.787073  1727 mvcc.cc:204] Tried to move back new op lower bound from 7317593128033837056 to 7317593127765536768. Current Snapshot: MvccSnapshot[applied={T|T < 7317593128033837056}]
I20260812 08:02:41.081125  1343 raft_consensus.cc:1275] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80 [term 1 FOLLOWER]: Refusing update from remote peer 8bcce3942e2e480191b4f6809e1377fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:41.081875  1721 consensus_queue.cc:1048] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e719176bf8b84aabbe7573da7d3a9d80" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:41.085314  1721 consensus_queue.cc:799] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd [LEADER]: Peer e719176bf8b84aabbe7573da7d3a9d80 is lagging by at least 36 ops behind the committed index 
I20260812 08:02:41.096643  1740 mvcc.cc:204] Tried to move back new op lower bound from 7317593128106004480 to 7317593127765536768. Current Snapshot: MvccSnapshot[applied={T|T < 7317593128033837056}]
I20260812 08:02:41.171000  1613 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:02:41.171433  1321 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260812 08:02:41.185330  1466 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 2e37a56783024e73beffff68156ee0b0 | 127.22.208.190:33325 | HEALTHY

Unusual flags for Master:
               Flag               |                                                                                                         Value                                                                                                          |      Tags       |         Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
 ipki_ca_key_size                 | 768                                                                                                                                                                                                                    | experimental    | all 1 server(s) checked
 ipki_server_key_size             | 768                                                                                                                                                                                                                    | experimental    | all 1 server(s) checked
 never_fsync                      | true                                                                                                                                                                                                                   | unsafe,advanced | all 1 server(s) checked
 openssl_security_level_override  | 0                                                                                                                                                                                                                      | unsafe,hidden   | all 1 server(s) checked
 rpc_reuseport                    | true                                                                                                                                                                                                                   | experimental    | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports | true                                                                                                                                                                                                                   | unsafe          | all 1 server(s) checked
 server_dump_info_format          | pb                                                                                                                                                                                                                     | hidden          | all 1 server(s) checked
 server_dump_info_path            | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden          | all 1 server(s) checked
 tsk_num_rsa_bits                 | 512                                                                                                                                                                                                                    | experimental    | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:36311 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 07e86a1b785c452a9a2f28d5bd3a70f5 | 127.22.208.131:32793 | HEALTHY | <none>   |       0        |       0
 8bcce3942e2e480191b4f6809e1377fd | 127.22.208.130:41643 | HEALTHY | <none>   |       1        |       0
 e719176bf8b84aabbe7573da7d3a9d80 | 127.22.208.129:38987 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Unusual flags for Tablet Server:
                  Flag                   |                                                                                                       Value                                                                                                        |         Tags         |      Tablet Server
-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
 flush_threshold_secs                    | 3                                                                                                                                                                                                                  | runtime,experimental | all 3 server(s) checked
 ipki_server_key_size                    | 768                                                                                                                                                                                                                | experimental         | all 3 server(s) checked
 local_ip_for_outbound_sockets           | 127.22.208.129                                                                                                                                                                                                     | experimental         | 127.22.208.129:38987
 local_ip_for_outbound_sockets           | 127.22.208.130                                                                                                                                                                                                     | experimental         | 127.22.208.130:41643
 local_ip_for_outbound_sockets           | 127.22.208.131                                                                                                                                                                                                     | experimental         | 127.22.208.131:32793
 log_compression_codec                   | no_compression                                                                                                                                                                                                     | experimental         | all 3 server(s) checked
 maintenance_manager_polling_interval_ms | 100                                                                                                                                                                                                                | hidden               | all 3 server(s) checked
 max_cell_size_bytes                     | 1000000                                                                                                                                                                                                            | unsafe               | all 3 server(s) checked
 never_fsync                             | true                                                                                                                                                                                                               | unsafe,advanced      | all 3 server(s) checked
 openssl_security_level_override         | 0                                                                                                                                                                                                                  | unsafe,hidden        | all 3 server(s) checked
 rpc_server_allow_ephemeral_ports        | true                                                                                                                                                                                                               | unsafe               | all 3 server(s) checked
 server_dump_info_format                 | pb                                                                                                                                                                                                                 | hidden               | all 3 server(s) checked
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden               | 127.22.208.129:38987
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden               | 127.22.208.130:41643
 server_dump_info_path                   | /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden               | 127.22.208.131:32793
 test_server_key                         | 000102030405060708090a0b0c0d0e2a                                                                                                                                                                                   | hidden               | all 3 server(s) checked
 test_server_key_iv                      | 2a0e0d0c0b0a09080706050403020100                                                                                                                                                                                   | hidden               | all 3 server(s) checked
 test_server_key_version                 | kudutenantkey@0                                                                                                                                                                                                    | hidden               | all 3 server(s) checked

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.22.208.148:36311 | all 3 server(s) checked
 time_source         | builtin              | all 3 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.19.0-SNAPSHOT | all 4 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
   Name    | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable | 3  | HEALTHY | 1             | 1       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1
 First Quartile | 1
 Median         | 1
 Third Quartile | 1
 Maximum        | 1

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 1
 Replicas       | 3

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20260812 08:02:41.220818 23362 log_verifier.cc:126] Checking tablet b8189424ca9e4aa7b21e9da93562acaf
I20260812 08:02:41.247581 23362 log_verifier.cc:177] Verified matching terms for 41 ops in tablet b8189424ca9e4aa7b21e9da93562acaf
I20260812 08:02:42.479245  1693 maintenance_manager.cc:419] P 07e86a1b785c452a9a2f28d5bd3a70f5: Scheduling FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf): perf score=0.005699
I20260812 08:02:42.487406  1396 maintenance_manager.cc:419] P e719176bf8b84aabbe7573da7d3a9d80: Scheduling FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf): perf score=0.005699
I20260812 08:02:42.541054  1544 maintenance_manager.cc:419] P 8bcce3942e2e480191b4f6809e1377fd: Scheduling FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf): perf score=0.005699
I20260812 08:02:42.562561  1567 maintenance_manager.cc:643] P 07e86a1b785c452a9a2f28d5bd3a70f5: FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.083s	user 0.078s	sys 0.000s Metrics: {"bytes_written":135745,"cfile_init":1,"compiler_manager_pool.queue_time_us":237,"dirs.queue_time_us":7134,"drs_written":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1681,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":40,"thread_start_us":645,"threads_started":3,"wal-append.queue_time_us":597}
I20260812 08:02:42.564469  1693 maintenance_manager.cc:419] P 07e86a1b785c452a9a2f28d5bd3a70f5: Scheduling LogGCOp(b8189424ca9e4aa7b21e9da93562acaf): free 4596811 bytes of WAL
I20260812 08:02:42.564905  1567 log_reader.cc:385] T b8189424ca9e4aa7b21e9da93562acaf: removed 5 log segments from log reader
I20260812 08:02:42.565276  1567 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000001 (ops 1-8)
I20260812 08:02:42.566020  1567 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000002 (ops 9-15)
I20260812 08:02:42.566207  1567 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000003 (ops 16-22)
I20260812 08:02:42.566316  1567 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000004 (ops 23-29)
I20260812 08:02:42.566406  1567 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 07e86a1b785c452a9a2f28d5bd3a70f5: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000005 (ops 30-36)
I20260812 08:02:42.569197  1567 maintenance_manager.cc:643] P 07e86a1b785c452a9a2f28d5bd3a70f5: LogGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.004s	user 0.001s	sys 0.000s Metrics: {}
I20260812 08:02:42.569733  1693 maintenance_manager.cc:419] P 07e86a1b785c452a9a2f28d5bd3a70f5: Scheduling UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf): 582 bytes on disk
I20260812 08:02:42.570611  1567 maintenance_manager.cc:643] P 07e86a1b785c452a9a2f28d5bd3a70f5: UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20260812 08:02:42.610571  1269 maintenance_manager.cc:643] P e719176bf8b84aabbe7573da7d3a9d80: FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.123s	user 0.078s	sys 0.000s Metrics: {"bytes_written":135745,"cfile_init":1,"compiler_manager_pool.queue_time_us":227,"dirs.queue_time_us":4992,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1277,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":40,"thread_start_us":347,"threads_started":3,"wal-append.queue_time_us":16900}
I20260812 08:02:42.612155  1396 maintenance_manager.cc:419] P e719176bf8b84aabbe7573da7d3a9d80: Scheduling LogGCOp(b8189424ca9e4aa7b21e9da93562acaf): free 4597064 bytes of WAL
I20260812 08:02:42.612555  1269 log_reader.cc:385] T b8189424ca9e4aa7b21e9da93562acaf: removed 8 log segments from log reader
I20260812 08:02:42.612677  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000001 (ops 1-5)
I20260812 08:02:42.612803  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000002 (ops 6-9)
I20260812 08:02:42.612892  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000003 (ops 10-13)
I20260812 08:02:42.613000  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000004 (ops 14-17)
I20260812 08:02:42.613101  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000005 (ops 18-21)
I20260812 08:02:42.613186  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000006 (ops 22-25)
I20260812 08:02:42.613271  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000007 (ops 26-29)
I20260812 08:02:42.613356  1269 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P e719176bf8b84aabbe7573da7d3a9d80: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000008 (ops 30-36)
I20260812 08:02:42.615945  1269 maintenance_manager.cc:643] P e719176bf8b84aabbe7573da7d3a9d80: LogGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.003s	user 0.000s	sys 0.002s Metrics: {}
I20260812 08:02:42.616636  1396 maintenance_manager.cc:419] P e719176bf8b84aabbe7573da7d3a9d80: Scheduling UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf): 582 bytes on disk
I20260812 08:02:42.617728  1269 maintenance_manager.cc:643] P e719176bf8b84aabbe7573da7d3a9d80: UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260812 08:02:42.644599  1417 maintenance_manager.cc:643] P 8bcce3942e2e480191b4f6809e1377fd: FlushMRSOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.103s	user 0.079s	sys 0.000s Metrics: {"bytes_written":135745,"cfile_init":1,"compiler_manager_pool.queue_time_us":28369,"dirs.queue_time_us":4781,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1372,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":40,"thread_start_us":351,"threads_started":3,"wal-append.queue_time_us":446}
I20260812 08:02:42.645896  1544 maintenance_manager.cc:419] P 8bcce3942e2e480191b4f6809e1377fd: Scheduling LogGCOp(b8189424ca9e4aa7b21e9da93562acaf): free 4596827 bytes of WAL
I20260812 08:02:42.646243  1417 log_reader.cc:385] T b8189424ca9e4aa7b21e9da93562acaf: removed 5 log segments from log reader
I20260812 08:02:42.646391  1417 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000001 (ops 1-8)
I20260812 08:02:42.646534  1417 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000002 (ops 9-15)
I20260812 08:02:42.646651  1417 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000003 (ops 16-22)
I20260812 08:02:42.646749  1417 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000004 (ops 23-29)
I20260812 08:02:42.647138  1417 log.cc:1079] T b8189424ca9e4aa7b21e9da93562acaf P 8bcce3942e2e480191b4f6809e1377fd: Deleting log segment in path: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.EncryptionEnabled_RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted_1.1786521625833775-23362-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8189424ca9e4aa7b21e9da93562acaf/wal-000000005 (ops 30-36)
I20260812 08:02:42.649080  1417 maintenance_manager.cc:643] P 8bcce3942e2e480191b4f6809e1377fd: LogGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.003s	user 0.001s	sys 0.000s Metrics: {}
I20260812 08:02:42.649600  1544 maintenance_manager.cc:419] P 8bcce3942e2e480191b4f6809e1377fd: Scheduling UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf): 582 bytes on disk
I20260812 08:02:42.650395  1417 maintenance_manager.cc:643] P 8bcce3942e2e480191b4f6809e1377fd: UndoDeltaBlockGCOp(b8189424ca9e4aa7b21e9da93562acaf) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20260812 08:02:43.437772 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1248
I20260812 08:02:43.448861 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1398
I20260812 08:02:43.459661 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1547
I20260812 08:02:43.476310 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1145
2026-08-12T08:02:43Z chronyd exiting
[       OK ] EncryptionEnabled/RaftConsensusParamEncryptionITest.TestCatchupAfterOpsEvicted/1 (5190 ms)
[----------] 1 test from EncryptionEnabled/RaftConsensusParamEncryptionITest (5190 ms total)

[----------] 1 test from RaftConsensusParamReplicationModesITest
[ RUN      ] RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID/1
2026-08-12T08:02:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:43Z Disabled control of system clock
I20260812 08:02:43.521314 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:38821
--webserver_interface=127.22.208.190
--webserver_port=0
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.208.190:38821
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 08:02:43.652915  1796 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:43.653208  1796 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:43.653254  1796 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:43.657436  1796 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:43.657541  1796 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:43.657564  1796 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:43.657585  1796 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:43.662760  1796 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.208.190:38821
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.208.190:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.208.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:43.664263  1796 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:43.665603  1796 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:43.672108  1804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.672123  1802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.672235  1801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:43.672487  1796 server_base.cc:1061] running on GCE node
I20260812 08:02:43.673044  1796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:43.674187  1796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:43.675385  1796 hybrid_clock.cc:648] HybridClock initialized: now 1786521763675367 us; error 35 us; skew 500 ppm
I20260812 08:02:43.677641  1796 webserver.cc:533] Webserver started at http://127.22.208.190:42079/ using document root <none> and password file <none>
I20260812 08:02:43.678413  1796 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:43.678494  1796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:43.678758  1796 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:43.680928  1796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data/instance:
uuid: "628b5e2925db48cc85ff003341a7d508"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:43.681591  1796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal/instance:
uuid: "628b5e2925db48cc85ff003341a7d508"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:43.686399  1796 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260812 08:02:43.689266  1810 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:43.690866  1796 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:43.691120  1796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal
uuid: "628b5e2925db48cc85ff003341a7d508"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:43.691293  1796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:43.706465  1796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:43.707343  1796 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:43.707578  1796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:43.717379  1796 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.190:38821
I20260812 08:02:43.717376  1870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.190:38821 every 8 connection(s)
I20260812 08:02:43.718542  1796 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:43.720472 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1796
I20260812 08:02:43.720618 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/master-0/wal/instance
I20260812 08:02:43.722124  1872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:43.729189  1872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: Bootstrap starting.
I20260812 08:02:43.732978  1872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:43.734181  1872 log.cc:826] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:43.736021  1872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: No bootstrap required, opened a new log
I20260812 08:02:43.739045  1872 raft_consensus.cc:359] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } }
I20260812 08:02:43.739343  1872 raft_consensus.cc:385] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:43.739408  1872 raft_consensus.cc:740] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628b5e2925db48cc85ff003341a7d508, State: Initialized, Role: FOLLOWER
I20260812 08:02:43.740000  1872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } }
I20260812 08:02:43.740191  1872 raft_consensus.cc:399] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:02:43.740262  1872 raft_consensus.cc:493] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:02:43.740377  1872 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:43.743912  1872 raft_consensus.cc:515] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } }
I20260812 08:02:43.744472  1872 leader_election.cc:304] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628b5e2925db48cc85ff003341a7d508; no voters: 
I20260812 08:02:43.744922  1872 leader_election.cc:290] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:02:43.745061  1877 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:43.745347  1877 raft_consensus.cc:697] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [term 1 LEADER]: Becoming Leader. State: Replica: 628b5e2925db48cc85ff003341a7d508, State: Running, Role: LEADER
I20260812 08:02:43.745721  1877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } }
I20260812 08:02:43.746718  1872 sys_catalog.cc:565] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:43.748476  1879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628b5e2925db48cc85ff003341a7d508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } } }
I20260812 08:02:43.748654  1879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:43.749090  1878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628b5e2925db48cc85ff003341a7d508. Latest consensus state: current_term: 1 leader_uuid: "628b5e2925db48cc85ff003341a7d508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b5e2925db48cc85ff003341a7d508" member_type: VOTER last_known_addr { host: "127.22.208.190" port: 38821 } } }
I20260812 08:02:43.749230  1878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508 [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:43.753062  1894 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:43.753150  1894 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:43.753249  1886 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:43.754104  1886 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:43.760406  1886 catalog_manager.cc:1383] Generated new cluster ID: 8dcae2c502e74ad594d124cabf7ebb99
I20260812 08:02:43.760502  1886 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:43.800995  1886 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:43.802002  1886 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:43.821276  1886 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 628b5e2925db48cc85ff003341a7d508: Generated new TSK 0
I20260812 08:02:43.822108  1886 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:43.828292 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:0
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 08:02:43.972501  1900 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:43.972743  1900 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:43.972788  1900 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:43.976334  1900 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:43.976466  1900 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:02:43.980568  1900 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--never_fsync=true
--heap_profile_path=/tmp/kudu.1900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:43.981709  1900 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:43.982924  1900 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:43.990056  1905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.990751  1908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.990751  1906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:43.991046  1900 server_base.cc:1061] running on GCE node
I20260812 08:02:43.991577  1900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:43.992200  1900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:43.993407  1900 hybrid_clock.cc:648] HybridClock initialized: now 1786521763993379 us; error 42 us; skew 500 ppm
I20260812 08:02:43.995479  1900 webserver.cc:533] Webserver started at http://127.22.208.129:35733/ using document root <none> and password file <none>
I20260812 08:02:43.996088  1900 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:43.996171  1900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:43.996351  1900 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:43.998317  1900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data/instance:
uuid: "3d5a93c6df6741aa876e18f7cc21185c"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:43.999007  1900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal/instance:
uuid: "3d5a93c6df6741aa876e18f7cc21185c"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:44.002844  1900 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:02:44.005537  1914 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.007061  1900 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:44.007185  1900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal
uuid: "3d5a93c6df6741aa876e18f7cc21185c"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-dhph"
I20260812 08:02:44.007344  1900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.062430  1900 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.063408  1900 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.063638  1900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.064291  1900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:44.065428  1900 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:44.065505  1900 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.065582  1900 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:44.065626  1900 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.078224  2042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:33759 every 8 connection(s)
I20260812 08:02:44.078174  1900 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:33759
I20260812 08:02:44.079741  1900 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:44.089746 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 1900
I20260812 08:02:44.089915 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal/instance
I20260812 08:02:44.090023  2043 heartbeater.cc:344] Connected to a master server at 127.22.208.190:38821
I20260812 08:02:44.090354  2043 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:44.091116  2043 heartbeater.cc:507] Master 127.22.208.190:38821 requested a full tablet report, sending...
I20260812 08:02:44.092914 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.130:0
--local_ip_for_outbound_sockets=127.22.208.130
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 08:02:44.092971  1827 ts_manager.cc:194] Registered new tserver with Master: 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:44.094671  1827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:59133
W20260812 08:02:44.237538  2046 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:44.237813  2046 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:44.237860  2046 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:44.242982  2046 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:44.243201  2046 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.130
I20260812 08:02:44.248837  2046 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.208.130
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--never_fsync=true
--heap_profile_path=/tmp/kudu.2046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.130

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:44.250909  2046 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:44.252451  2046 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:44.262981  2055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.263134  2054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.263402  2046 server_base.cc:1061] running on GCE node
W20260812 08:02:44.263448  2057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.264147  2046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:44.264804  2046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:44.266217  2046 hybrid_clock.cc:648] HybridClock initialized: now 1786521764266138 us; error 99 us; skew 500 ppm
I20260812 08:02:44.268607  2046 webserver.cc:533] Webserver started at http://127.22.208.130:43213/ using document root <none> and password file <none>
I20260812 08:02:44.269407  2046 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:44.269477  2046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:44.269773  2046 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:44.271742  2046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data/instance:
uuid: "714a507a0a7149e6a58104455bec51f2"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.272657  2046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal/instance:
uuid: "714a507a0a7149e6a58104455bec51f2"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.277287  2046 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260812 08:02:44.280454  2063 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.281888  2046 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260812 08:02:44.282064  2046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal
uuid: "714a507a0a7149e6a58104455bec51f2"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.282208  2046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.295087  2046 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.295861  2046 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.296451  2046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.298285  2046 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:44.299638  2046 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:44.299741  2046 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.299844  2046 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:44.299906  2046 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.314568  2046 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.130:44137
I20260812 08:02:44.314680  2186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.130:44137 every 8 connection(s)
I20260812 08:02:44.315860  2046 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/data/info.pb
I20260812 08:02:44.324096 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 2046
I20260812 08:02:44.325038 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-1/wal/instance
I20260812 08:02:44.328034  2187 heartbeater.cc:344] Connected to a master server at 127.22.208.190:38821
I20260812 08:02:44.328415  2187 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:44.328912 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.131:0
--local_ip_for_outbound_sockets=127.22.208.131
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 08:02:44.329103  2187 heartbeater.cc:507] Master 127.22.208.190:38821 requested a full tablet report, sending...
I20260812 08:02:44.330539  1827 ts_manager.cc:194] Registered new tserver with Master: 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137)
I20260812 08:02:44.331633  1827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.130:49205
W20260812 08:02:44.465557  2192 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:44.465814  2192 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:44.465859  2192 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:44.470736  2192 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:44.470886  2192 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.131
I20260812 08:02:44.475521  2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.208.131
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.131

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:44.476784  2192 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:44.478081  2192 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:44.485908  2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.486057  2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.486173  2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.486500  2192 server_base.cc:1061] running on GCE node
I20260812 08:02:44.487079  2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:44.487726  2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:44.488966  2192 hybrid_clock.cc:648] HybridClock initialized: now 1786521764488887 us; error 93 us; skew 500 ppm
I20260812 08:02:44.492177  2192 webserver.cc:533] Webserver started at http://127.22.208.131:44379/ using document root <none> and password file <none>
I20260812 08:02:44.492898  2192 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:44.492974  2192 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:44.493382  2192 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:44.495580  2192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data/instance:
uuid: "1d233039be434b58822feb9300ddbbd8"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.496207  2192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal/instance:
uuid: "1d233039be434b58822feb9300ddbbd8"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.500945  2192 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:44.504384  2208 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.506347  2192 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:02:44.506572  2192 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal
uuid: "1d233039be434b58822feb9300ddbbd8"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.506793  2192 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.520774  2192 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.521576  2192 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.521822  2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.522478  2192 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:44.523685  2192 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:44.523777  2192 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.523861  2192 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:44.523911  2192 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.538067  2192 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.131:39265
I20260812 08:02:44.538133  2336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.131:39265 every 8 connection(s)
I20260812 08:02:44.539211  2192 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/data/info.pb
I20260812 08:02:44.549470 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 2192
I20260812 08:02:44.549701 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-2/wal/instance
I20260812 08:02:44.550956  2337 heartbeater.cc:344] Connected to a master server at 127.22.208.190:38821
I20260812 08:02:44.551231  2337 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:44.551796  2337 heartbeater.cc:507] Master 127.22.208.190:38821 requested a full tablet report, sending...
I20260812 08:02:44.552186 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.132:0
--local_ip_for_outbound_sockets=127.22.208.132
--webserver_interface=127.22.208.132
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--raft_prepare_replacement_before_eviction=true with env {}
I20260812 08:02:44.552994  1827 ts_manager.cc:194] Registered new tserver with Master: 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265)
I20260812 08:02:44.553874  1827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.131:38721
W20260812 08:02:44.710768  2340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:44.711115  2340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:44.711223  2340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:44.715690  2340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:44.715842  2340 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.132
I20260812 08:02:44.722412  2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.22.208.132
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--never_fsync=true
--heap_profile_path=/tmp/kudu.2340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.132

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:44.723681  2340 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:44.725107  2340 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:44.734398  2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.734401  2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.734428  2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.735200  2340 server_base.cc:1061] running on GCE node
I20260812 08:02:44.735780  2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:44.736474  2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:44.737710  2340 hybrid_clock.cc:648] HybridClock initialized: now 1786521764737667 us; error 55 us; skew 500 ppm
I20260812 08:02:44.740420  2340 webserver.cc:533] Webserver started at http://127.22.208.132:40855/ using document root <none> and password file <none>
I20260812 08:02:44.741202  2340 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:44.741333  2340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:44.741693  2340 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:44.744042  2340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data/instance:
uuid: "1f20f95d7b6d4556ab254e8ed6506085"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.744637  2340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal/instance:
uuid: "1f20f95d7b6d4556ab254e8ed6506085"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.749259  2340 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:44.752663  2358 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.754354  2340 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:02:44.754488  2340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal
uuid: "1f20f95d7b6d4556ab254e8ed6506085"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-dhph"
I20260812 08:02:44.754676  2340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.768913  2340 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.769623  2340 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.769930  2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.770684  2340 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:44.772029  2340 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:44.772110  2340 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.772184  2340 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:44.772230  2340 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.786839  2340 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.132:46427
I20260812 08:02:44.786924  2479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.132:46427 every 8 connection(s)
I20260812 08:02:44.787963  2340 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/data/info.pb
I20260812 08:02:44.793193 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 2340
I20260812 08:02:44.793375 23362 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-3/wal/instance
I20260812 08:02:44.798492  2480 heartbeater.cc:344] Connected to a master server at 127.22.208.190:38821
I20260812 08:02:44.798890  2480 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:44.799522  2480 heartbeater.cc:507] Master 127.22.208.190:38821 requested a full tablet report, sending...
I20260812 08:02:44.800806  1827 ts_manager.cc:194] Registered new tserver with Master: 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:44.801385  1827 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.132:36713
I20260812 08:02:44.810070 23362 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20260812 08:02:44.824882  1827 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45416:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
  indirect_data: "<redacted>"""
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260812 08:02:44.852456  2262 tablet_service.cc:1511] Processing CreateTablet for tablet 7f94f84d271246aa9aaef39ef0c7eaac (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260812 08:02:44.853708  2259 tablet_service.cc:1511] Processing CreateTablet for tablet 071986b2c7af4287bb4a2e6063c28a69 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260812 08:02:44.854146  2262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f94f84d271246aa9aaef39ef0c7eaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.854455  2261 tablet_service.cc:1511] Processing CreateTablet for tablet 67e4cbd0781149a08069d389705a2643 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260812 08:02:44.855211  2261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e4cbd0781149a08069d389705a2643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.856298  2259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071986b2c7af4287bb4a2e6063c28a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.856228  2260 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecf598b966e4dcabcd62a0ba48180aa (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260812 08:02:44.856501  2260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecf598b966e4dcabcd62a0ba48180aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.856485  2115 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecf598b966e4dcabcd62a0ba48180aa (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260812 08:02:44.857818  2115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecf598b966e4dcabcd62a0ba48180aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.858990  2114 tablet_service.cc:1511] Processing CreateTablet for tablet b5224a5c2c4a471e9d3be65b16da7189 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260812 08:02:44.859428  2114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5224a5c2c4a471e9d3be65b16da7189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.859133  1963 tablet_service.cc:1511] Processing CreateTablet for tablet 071986b2c7af4287bb4a2e6063c28a69 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260812 08:02:44.860450  1965 tablet_service.cc:1511] Processing CreateTablet for tablet 3ecf598b966e4dcabcd62a0ba48180aa (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260812 08:02:44.861078  1966 tablet_service.cc:1511] Processing CreateTablet for tablet 7f94f84d271246aa9aaef39ef0c7eaac (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260812 08:02:44.862905  2112 tablet_service.cc:1511] Processing CreateTablet for tablet 67e4cbd0781149a08069d389705a2643 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260812 08:02:44.863224  2112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e4cbd0781149a08069d389705a2643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.863265  1966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f94f84d271246aa9aaef39ef0c7eaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.863401  2508 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: Bootstrap starting.
I20260812 08:02:44.860108  1964 tablet_service.cc:1511] Processing CreateTablet for tablet b5224a5c2c4a471e9d3be65b16da7189 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260812 08:02:44.865042  1964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5224a5c2c4a471e9d3be65b16da7189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.865633  1965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ecf598b966e4dcabcd62a0ba48180aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.866472  2508 tablet_bootstrap.cc:654] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.866945  2109 tablet_service.cc:1511] Processing CreateTablet for tablet 071986b2c7af4287bb4a2e6063c28a69 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260812 08:02:44.867303  2109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071986b2c7af4287bb4a2e6063c28a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.867882  2508 log.cc:826] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.870071  2508 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: No bootstrap required, opened a new log
I20260812 08:02:44.870312  2508 ts_tablet_manager.cc:1403] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20260812 08:02:44.872514  1963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071986b2c7af4287bb4a2e6063c28a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.872742  2510 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: Bootstrap starting.
I20260812 08:02:44.873836  2508 raft_consensus.cc:359] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.874084  2508 raft_consensus.cc:385] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.874136  2508 raft_consensus.cc:740] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.874702  2508 consensus_queue.cc:260] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.873867  2407 tablet_service.cc:1511] Processing CreateTablet for tablet b5224a5c2c4a471e9d3be65b16da7189 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260812 08:02:44.875718  2508 ts_tablet_manager.cc:1434] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260812 08:02:44.875765  2510 tablet_bootstrap.cc:654] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.876091  2337 heartbeater.cc:499] Master 127.22.208.190:38821 was elected leader, sending a full tablet report...
I20260812 08:02:44.876188  2407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5224a5c2c4a471e9d3be65b16da7189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.876745  2510 log.cc:826] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.877502  2508 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8: Bootstrap starting.
I20260812 08:02:44.875402  2410 tablet_service.cc:1511] Processing CreateTablet for tablet 7f94f84d271246aa9aaef39ef0c7eaac (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260812 08:02:44.878304  2410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f94f84d271246aa9aaef39ef0c7eaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.879093  2508 tablet_bootstrap.cc:654] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.874909  2409 tablet_service.cc:1511] Processing CreateTablet for tablet 67e4cbd0781149a08069d389705a2643 (DEFAULT_TABLE table=test-workload [id=3149907d56be4a1294f29fe93c26fa13]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260812 08:02:44.880455  2409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e4cbd0781149a08069d389705a2643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.881723  2508 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8: No bootstrap required, opened a new log
I20260812 08:02:44.881820  2508 ts_tablet_manager.cc:1403] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:02:44.882547  2508 raft_consensus.cc:359] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.882750  2508 raft_consensus.cc:385] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.882800  2508 raft_consensus.cc:740] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.882965  2508 consensus_queue.cc:260] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.883314  2508 ts_tablet_manager.cc:1434] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.001s
I20260812 08:02:44.883539  2508 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8: Bootstrap starting.
I20260812 08:02:44.884886  2508 tablet_bootstrap.cc:654] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.885766  2514 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Bootstrap starting.
I20260812 08:02:44.887142  2508 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8: No bootstrap required, opened a new log
I20260812 08:02:44.887252  2508 ts_tablet_manager.cc:1403] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20260812 08:02:44.887813  2508 raft_consensus.cc:359] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.887955  2508 raft_consensus.cc:385] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.888010  2508 raft_consensus.cc:740] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.888175  2508 consensus_queue.cc:260] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.888574  2508 ts_tablet_manager.cc:1434] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:02:44.888711  2515 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: Bootstrap starting.
I20260812 08:02:44.888826  2508 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: Bootstrap starting.
I20260812 08:02:44.889292  2510 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: No bootstrap required, opened a new log
I20260812 08:02:44.889541  2510 ts_tablet_manager.cc:1403] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I20260812 08:02:44.890287  2508 tablet_bootstrap.cc:654] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.892293  2514 tablet_bootstrap.cc:654] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.892582  2508 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: No bootstrap required, opened a new log
I20260812 08:02:44.892696  2508 ts_tablet_manager.cc:1403] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20260812 08:02:44.893303  2508 raft_consensus.cc:359] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.893445  2508 raft_consensus.cc:385] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.893500  2508 raft_consensus.cc:740] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.893671  2508 consensus_queue.cc:260] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.893744  2515 tablet_bootstrap.cc:654] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.894025  2508 ts_tablet_manager.cc:1434] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20260812 08:02:44.893884  2510 raft_consensus.cc:359] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.894141  2510 raft_consensus.cc:385] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.894224  2510 raft_consensus.cc:740] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.894845  2515 log.cc:826] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.894827  2510 consensus_queue.cc:260] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.895407  2514 log.cc:826] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.895905  2510 ts_tablet_manager.cc:1434] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260812 08:02:44.896026  2043 heartbeater.cc:499] Master 127.22.208.190:38821 was elected leader, sending a full tablet report...
I20260812 08:02:44.896216  2510 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c: Bootstrap starting.
I20260812 08:02:44.897667  2514 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: No bootstrap required, opened a new log
I20260812 08:02:44.897919  2510 tablet_bootstrap.cc:654] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.897923  2514 ts_tablet_manager.cc:1403] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.003s
I20260812 08:02:44.899544  2515 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: No bootstrap required, opened a new log
I20260812 08:02:44.899605  2510 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c: No bootstrap required, opened a new log
I20260812 08:02:44.899693  2510 ts_tablet_manager.cc:1403] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20260812 08:02:44.899887  2515 ts_tablet_manager.cc:1403] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.003s
I20260812 08:02:44.900370  2510 raft_consensus.cc:359] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.900504  2510 raft_consensus.cc:385] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.900540  2510 raft_consensus.cc:740] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.900693  2510 consensus_queue.cc:260] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.901000  2510 ts_tablet_manager.cc:1434] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20260812 08:02:44.901221  2510 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c: Bootstrap starting.
I20260812 08:02:44.902308  2514 raft_consensus.cc:359] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.902540  2510 tablet_bootstrap.cc:654] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.902559  2514 raft_consensus.cc:385] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.902688  2514 raft_consensus.cc:740] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.903190  2514 consensus_queue.cc:260] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.904227  2514 ts_tablet_manager.cc:1434] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.004s
I20260812 08:02:44.904784  2187 heartbeater.cc:499] Master 127.22.208.190:38821 was elected leader, sending a full tablet report...
I20260812 08:02:44.907011  2510 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c: No bootstrap required, opened a new log
I20260812 08:02:44.907143  2510 ts_tablet_manager.cc:1403] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20260812 08:02:44.907070  2515 raft_consensus.cc:359] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.907447  2515 raft_consensus.cc:385] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.907526  2515 raft_consensus.cc:740] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f20f95d7b6d4556ab254e8ed6506085, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.907577  2514 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2: Bootstrap starting.
I20260812 08:02:44.908413  2510 raft_consensus.cc:359] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.908548  2510 raft_consensus.cc:385] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.908586  2510 raft_consensus.cc:740] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.909333  2514 tablet_bootstrap.cc:654] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.909543  2510 consensus_queue.cc:260] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.909564  2515 consensus_queue.cc:260] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.910254  2510 ts_tablet_manager.cc:1434] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:44.910543  2510 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: Bootstrap starting.
I20260812 08:02:44.910908  2480 heartbeater.cc:499] Master 127.22.208.190:38821 was elected leader, sending a full tablet report...
I20260812 08:02:44.911738  2515 ts_tablet_manager.cc:1434] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.002s
I20260812 08:02:44.912258  2515 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: Bootstrap starting.
I20260812 08:02:44.912883  2514 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2: No bootstrap required, opened a new log
I20260812 08:02:44.912982  2510 tablet_bootstrap.cc:654] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.913094  2514 ts_tablet_manager.cc:1403] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20260812 08:02:44.914011  2515 tablet_bootstrap.cc:654] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.914414  2514 raft_consensus.cc:359] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.914603  2514 raft_consensus.cc:385] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.914662  2514 raft_consensus.cc:740] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.914885  2510 tablet_bootstrap.cc:492] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: No bootstrap required, opened a new log
I20260812 08:02:44.914985  2510 ts_tablet_manager.cc:1403] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20260812 08:02:44.915035  2514 consensus_queue.cc:260] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.915369  2515 tablet_bootstrap.cc:492] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: No bootstrap required, opened a new log
I20260812 08:02:44.915458  2515 ts_tablet_manager.cc:1403] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:02:44.915464  2514 ts_tablet_manager.cc:1434] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:44.915565  2510 raft_consensus.cc:359] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.915690  2510 raft_consensus.cc:385] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.915726  2510 raft_consensus.cc:740] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.915922  2510 consensus_queue.cc:260] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.916069  2514 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2: Bootstrap starting.
I20260812 08:02:44.916165  2515 raft_consensus.cc:359] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.916289  2515 raft_consensus.cc:385] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.916327  2515 raft_consensus.cc:740] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f20f95d7b6d4556ab254e8ed6506085, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.916375  2510 ts_tablet_manager.cc:1434] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:02:44.916472  2515 consensus_queue.cc:260] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.916973  2515 ts_tablet_manager.cc:1434] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20260812 08:02:44.917183  2515 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: Bootstrap starting.
I20260812 08:02:44.917855  2514 tablet_bootstrap.cc:654] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.918929  2515 tablet_bootstrap.cc:654] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.919056  2511 raft_consensus.cc:493] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:44.919272  2511 raft_consensus.cc:515] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.920557  2514 tablet_bootstrap.cc:492] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2: No bootstrap required, opened a new log
I20260812 08:02:44.920696  2514 ts_tablet_manager.cc:1403] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20260812 08:02:44.921926  2514 raft_consensus.cc:359] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.922077  2514 raft_consensus.cc:385] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.922132  2514 raft_consensus.cc:740] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.922292  2514 consensus_queue.cc:260] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:44.922683  2514 ts_tablet_manager.cc:1434] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:02:44.922922  2514 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2: Bootstrap starting.
I20260812 08:02:44.922874  2511 leader_election.cc:290] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:44.924906  2514 tablet_bootstrap.cc:654] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.925822  2515 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: No bootstrap required, opened a new log
I20260812 08:02:44.926244  2518 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:44.926473  2518 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.928088  2514 tablet_bootstrap.cc:492] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2: No bootstrap required, opened a new log
I20260812 08:02:44.928195  2514 ts_tablet_manager.cc:1403] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20260812 08:02:44.929033  2518 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265)
I20260812 08:02:44.929481  2514 raft_consensus.cc:359] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.929697  2515 ts_tablet_manager.cc:1403] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.001s
I20260812 08:02:44.929800  2514 raft_consensus.cc:385] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.929888  2514 raft_consensus.cc:740] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.930084  2514 consensus_queue.cc:260] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.930483  2514 ts_tablet_manager.cc:1434] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:02:44.930733  2515 raft_consensus.cc:359] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.930866  2515 raft_consensus.cc:385] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.930903  2515 raft_consensus.cc:740] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f20f95d7b6d4556ab254e8ed6506085, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.931046  2515 consensus_queue.cc:260] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:44.932377  1990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" is_pre_election: true
I20260812 08:02:44.932781  1990 raft_consensus.cc:2468] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 0.
I20260812 08:02:44.933727  2515 ts_tablet_manager.cc:1434] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20260812 08:02:44.936890  2211 leader_election.cc:304] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 3d5a93c6df6741aa876e18f7cc21185c; no voters: 
I20260812 08:02:44.937218  2511 raft_consensus.cc:2804] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:44.937069  2138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
I20260812 08:02:44.937404  2511 raft_consensus.cc:493] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:44.937485  2511 raft_consensus.cc:3060] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.937618  2138 raft_consensus.cc:2468] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 0.
I20260812 08:02:44.939335  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8" is_pre_election: true
I20260812 08:02:44.939852  2278 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 0.
I20260812 08:02:44.940192  2511 raft_consensus.cc:515] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.940999  1918 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 3d5a93c6df6741aa876e18f7cc21185c; no voters: 
I20260812 08:02:44.941155  2511 leader_election.cc:290] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 1 election: Requested vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:44.941433  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
I20260812 08:02:44.941491  2518 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:44.941433  2138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2"
I20260812 08:02:44.941646  2135 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 0.
I20260812 08:02:44.941710  2138 raft_consensus.cc:3060] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.941583  2518 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:44.941893  2518 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.942014  1990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c"
I20260812 08:02:44.942126  1990 raft_consensus.cc:3060] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.943158  2518 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.943239  1990 raft_consensus.cc:2468] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 1.
I20260812 08:02:44.943240  2138 raft_consensus.cc:2468] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 1.
I20260812 08:02:44.943619  2518 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 election: Requested vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265)
I20260812 08:02:44.943609  2211 leader_election.cc:304] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 3d5a93c6df6741aa876e18f7cc21185c; no voters: 
I20260812 08:02:44.943931  2138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2"
I20260812 08:02:44.943979  2511 raft_consensus.cc:2804] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:44.944073  2138 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.944449  2511 raft_consensus.cc:697] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Running, Role: LEADER
I20260812 08:02:44.944557  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8"
I20260812 08:02:44.945029  2278 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.945040  2511 consensus_queue.cc:237] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.945909  2138 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 1.
I20260812 08:02:44.946314  1916 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5a93c6df6741aa876e18f7cc21185c, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:44.946554  2518 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:44.946767  2278 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 1.
I20260812 08:02:44.946858  2518 raft_consensus.cc:697] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 LEADER]: Becoming Leader. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Running, Role: LEADER
I20260812 08:02:44.947467  2518 consensus_queue.cc:237] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:44.949959  1827 catalog_manager.cc:5719] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d233039be434b58822feb9300ddbbd8 (127.22.208.131). New cstate: current_term: 1 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:44.951936  1832 catalog_manager.cc:5719] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129). New cstate: current_term: 1 leader_uuid: "3d5a93c6df6741aa876e18f7cc21185c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:45.021018  2526 raft_consensus.cc:493] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:45.021309  2526 raft_consensus.cc:515] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.022568  2526 leader_election.cc:290] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:45.026513  2523 raft_consensus.cc:493] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:45.026767  2523 raft_consensus.cc:515] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.027391  1990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" candidate_uuid: "1f20f95d7b6d4556ab254e8ed6506085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" is_pre_election: true
I20260812 08:02:45.027560  1990 raft_consensus.cc:2468] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f20f95d7b6d4556ab254e8ed6506085 in term 0.
I20260812 08:02:45.028117  2362 leader_election.cc:304] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085, 3d5a93c6df6741aa876e18f7cc21185c; no voters: 
I20260812 08:02:45.028388  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" candidate_uuid: "1f20f95d7b6d4556ab254e8ed6506085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8" is_pre_election: true
I20260812 08:02:45.028450  2526 raft_consensus.cc:2804] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:45.028543  2526 raft_consensus.cc:493] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:45.028539  2523 leader_election.cc:290] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265), 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:45.028555  2278 raft_consensus.cc:2468] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f20f95d7b6d4556ab254e8ed6506085 in term 0.
I20260812 08:02:45.028610  2526 raft_consensus.cc:3060] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.029752  2526 raft_consensus.cc:515] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.030499  2526 leader_election.cc:290] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 1 election: Requested vote from peers 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:45.031039  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" candidate_uuid: "1f20f95d7b6d4556ab254e8ed6506085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8"
I20260812 08:02:45.031054  1990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" candidate_uuid: "1f20f95d7b6d4556ab254e8ed6506085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c"
I20260812 08:02:45.031365  1990 raft_consensus.cc:3060] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.031365  2278 raft_consensus.cc:3060] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.032645  1990 raft_consensus.cc:2468] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f20f95d7b6d4556ab254e8ed6506085 in term 1.
I20260812 08:02:45.032645  2278 raft_consensus.cc:2468] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f20f95d7b6d4556ab254e8ed6506085 in term 1.
I20260812 08:02:45.033111  2362 leader_election.cc:304] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085, 3d5a93c6df6741aa876e18f7cc21185c; no voters: 
I20260812 08:02:45.033434  2526 raft_consensus.cc:2804] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:45.033560  2526 raft_consensus.cc:697] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 LEADER]: Becoming Leader. State: Replica: 1f20f95d7b6d4556ab254e8ed6506085, State: Running, Role: LEADER
I20260812 08:02:45.033941  2526 consensus_queue.cc:237] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.035065  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67e4cbd0781149a08069d389705a2643" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8" is_pre_election: true
I20260812 08:02:45.035248  2278 raft_consensus.cc:2468] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 0.
I20260812 08:02:45.035386  2431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67e4cbd0781149a08069d389705a2643" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085" is_pre_election: true
I20260812 08:02:45.035591  2431 raft_consensus.cc:2468] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 0.
I20260812 08:02:45.035696  2067 leader_election.cc:304] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:45.036119  2523 raft_consensus.cc:2804] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:45.036196  2523 raft_consensus.cc:493] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:45.036238  2523 raft_consensus.cc:3060] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.037382  2523 raft_consensus.cc:515] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.037810  2523 leader_election.cc:290] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 1 election: Requested vote from peers 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265), 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:45.038213  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67e4cbd0781149a08069d389705a2643" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8"
I20260812 08:02:45.038358  2278 raft_consensus.cc:3060] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.038331  2431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67e4cbd0781149a08069d389705a2643" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085"
I20260812 08:02:45.038446  2431 raft_consensus.cc:3060] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Advancing to term 1
W20260812 08:02:45.039076  2481 tablet.cc:2406] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:45.039556  2278 raft_consensus.cc:2468] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 1.
I20260812 08:02:45.039599  2431 raft_consensus.cc:2468] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 1.
I20260812 08:02:45.040200  2065 leader_election.cc:304] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085, 714a507a0a7149e6a58104455bec51f2; no voters: 
W20260812 08:02:45.040493  2338 tablet.cc:2406] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:45.040562  2523 raft_consensus.cc:2804] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:45.041020  2523 raft_consensus.cc:697] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [term 1 LEADER]: Becoming Leader. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Running, Role: LEADER
I20260812 08:02:45.040922  1832 catalog_manager.cc:5719] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132). New cstate: current_term: 1 leader_uuid: "1f20f95d7b6d4556ab254e8ed6506085" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:45.041422  2523 consensus_queue.cc:237] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:45.045324  1832 catalog_manager.cc:5719] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 714a507a0a7149e6a58104455bec51f2 (127.22.208.130). New cstate: current_term: 1 leader_uuid: "714a507a0a7149e6a58104455bec51f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:45.067957  2189 tablet.cc:2406] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:02:45.081805  2044 tablet.cc:2406] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:45.095834  2538 raft_consensus.cc:493] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:45.095992  2538 raft_consensus.cc:515] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:45.096799  2538 leader_election.cc:290] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:45.097441  2138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
I20260812 08:02:45.097622  2138 raft_consensus.cc:2468] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 0.
I20260812 08:02:45.098166  1916 leader_election.cc:304] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d5a93c6df6741aa876e18f7cc21185c, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:45.098415  2538 raft_consensus.cc:2804] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:45.098510  2538 raft_consensus.cc:493] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:45.098609  2538 raft_consensus.cc:3060] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.100179  2538 raft_consensus.cc:515] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:45.100660  2538 leader_election.cc:290] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 election: Requested vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:45.101143  2138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2"
I20260812 08:02:45.101274  2138 raft_consensus.cc:3060] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.102566  2138 raft_consensus.cc:2468] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 1.
I20260812 08:02:45.102880  2431 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085" is_pre_election: true
I20260812 08:02:45.102934  2430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "3d5a93c6df6741aa876e18f7cc21185c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085"
I20260812 08:02:45.103152  2431 raft_consensus.cc:2468] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d5a93c6df6741aa876e18f7cc21185c in term 0.
I20260812 08:02:45.103832  1916 leader_election.cc:304] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d5a93c6df6741aa876e18f7cc21185c, 714a507a0a7149e6a58104455bec51f2; no voters: 1f20f95d7b6d4556ab254e8ed6506085
I20260812 08:02:45.104055  2538 raft_consensus.cc:2804] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:45.104209  2538 raft_consensus.cc:697] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [term 1 LEADER]: Becoming Leader. State: Replica: 3d5a93c6df6741aa876e18f7cc21185c, State: Running, Role: LEADER
I20260812 08:02:45.104382  2538 consensus_queue.cc:237] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:45.106877  1832 catalog_manager.cc:5719] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129). New cstate: current_term: 1 leader_uuid: "3d5a93c6df6741aa876e18f7cc21185c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:45.196799  1990 raft_consensus.cc:1275] T 3ecf598b966e4dcabcd62a0ba48180aa P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.197652  2138 raft_consensus.cc:1275] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.198501  2430 raft_consensus.cc:1275] T 67e4cbd0781149a08069d389705a2643 P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Refusing update from remote peer 714a507a0a7149e6a58104455bec51f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.198483  2511 consensus_queue.cc:1048] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.199543  2523 consensus_queue.cc:1048] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.199687  2278 raft_consensus.cc:1275] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Refusing update from remote peer 1f20f95d7b6d4556ab254e8ed6506085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.200320  2526 consensus_queue.cc:1048] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.202301  1990 raft_consensus.cc:1275] T 7f94f84d271246aa9aaef39ef0c7eaac P 3d5a93c6df6741aa876e18f7cc21185c [term 1 FOLLOWER]: Refusing update from remote peer 1f20f95d7b6d4556ab254e8ed6506085: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.202865  2526 consensus_queue.cc:1048] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.204651  2135 raft_consensus.cc:1275] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Refusing update from remote peer 3d5a93c6df6741aa876e18f7cc21185c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.205890  2283 raft_consensus.cc:1275] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Refusing update from remote peer 3d5a93c6df6741aa876e18f7cc21185c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.206503  2538 consensus_queue.cc:1048] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.206666  2281 raft_consensus.cc:1275] T 67e4cbd0781149a08069d389705a2643 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Refusing update from remote peer 714a507a0a7149e6a58104455bec51f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.210263  2431 raft_consensus.cc:3060] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:45.210948  2547 consensus_queue.cc:1048] T 67e4cbd0781149a08069d389705a2643 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.211591  2431 raft_consensus.cc:1275] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Refusing update from remote peer 3d5a93c6df6741aa876e18f7cc21185c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.212807  2511 consensus_queue.cc:1048] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.213792  2538 consensus_queue.cc:1048] T 071986b2c7af4287bb4a2e6063c28a69 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.214793  2559 mvcc.cc:204] Tried to move back new op lower bound from 7317593150250663936 to 7317593149220204544. Current Snapshot: MvccSnapshot[applied={T|T < 7317593150230994944}]
I20260812 08:02:45.217550  2137 raft_consensus.cc:1275] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Refusing update from remote peer 3d5a93c6df6741aa876e18f7cc21185c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:45.218153  2518 consensus_queue.cc:1048] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.218526  2518 consensus_queue.cc:1048] T b5224a5c2c4a471e9d3be65b16da7189 P 3d5a93c6df6741aa876e18f7cc21185c [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:45.225499  2563 mvcc.cc:204] Tried to move back new op lower bound from 7317593150250663936 to 7317593149220204544. Current Snapshot: MvccSnapshot[applied={T|T < 7317593150230994944}]
I20260812 08:02:45.226665  2562 mvcc.cc:204] Tried to move back new op lower bound from 7317593150261886976 to 7317593149229424640. Current Snapshot: MvccSnapshot[applied={T|T < 7317593150261886976}]
I20260812 08:02:45.238737  2560 mvcc.cc:204] Tried to move back new op lower bound from 7317593150244569088 to 7317593149613309952. Current Snapshot: MvccSnapshot[applied={T|T < 7317593150244569088}]
I20260812 08:02:45.676139 23362 raft_consensus-itest.cc:3120] Storing port 127.22.208.129:33759 for use by new tablet server
I20260812 08:02:45.676298 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1900
W20260812 08:02:45.694410  2486 connection.cc:573] client connection to 127.22.208.129:33759 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.694537  2486 meta_cache.cc:302] tablet b5224a5c2c4a471e9d3be65b16da7189: replica 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.694991  2362 connection.cc:573] client connection to 127.22.208.129:33759 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.695102  2211 connection.cc:573] client connection to 127.22.208.129:33759 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.695175  2362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.695257  2211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.695658  2211 connection.cc:573] server connection from 127.22.208.129:40833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 08:02:45.696995  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.697007  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.700249  2091 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.701001  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.701496  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.704823  2362 consensus_peers.cc:597] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:45.706045  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.706784  2389 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.707481  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.708643  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.712383  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.712563  2211 consensus_peers.cc:597] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:45.713053  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.712435  2239 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.714922  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.721470  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.724884  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.726963  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.728286  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.729719  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.733445  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.733963  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.736629  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.736846  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.738294  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.742406  2239 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.742406  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.742977  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.747853  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.756428  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.761154  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.761439  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.768118  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.770933  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.772099  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.774561  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.775969  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.780766  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.785725  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.786468  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.794313  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.798745  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.799005  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.800488  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.803534  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.808045  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.814538  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.814538  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.827272  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.831283  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.832818  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.838491  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.839428  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.845815  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.849373  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.849411  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.854140  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.857846  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.868547  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.868547  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.870347  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.885968  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.891065  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.897886  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.903561  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.905354  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.909112  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.912684  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.920868  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.925469  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.925469  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.926975  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.933353  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.948400  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.957638  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.958994  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.970247  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.979867  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.983012  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.983337  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.993943  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.994494  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:45.994714  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.007328  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.018903  2238 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.019243  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.036334  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.044186  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.046382  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.046655  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.060576  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.063889  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.063890  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.072409  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.074013  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.102989  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.103482  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.121965  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.122090  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.129528  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.132006  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.133524  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.135255  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.148340  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.150379  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.166214  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.183456  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.191172  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.210454  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.214308  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.216044  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.226028  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.227864  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.227880  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.245026  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.245121  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.260386  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.281683  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.292959  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.295274  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.314484  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.317775  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.324818  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.330715  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.350950  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.351716  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.365435  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.386739  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.394918  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.401922  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.403836  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.404918  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.425693  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.430301  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.432960  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.441450  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.441493  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.463258  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.482913  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.503791  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.507899  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.520135  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.520543  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.522468  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.543792  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.544426  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.550177  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.586524  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.596186  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.600311  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.600409  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.630259  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.641947  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.642120  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.673233  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.675570  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.684978  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.685204  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.689079  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.720059  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.721992  2486 meta_cache.cc:302] tablet 071986b2c7af4287bb4a2e6063c28a69: replica 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
W20260812 08:02:46.730849  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.734691  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.735976  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.775713  2386 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.775868  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.779301  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.812356  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.816274  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.827220  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.861828  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.863042  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.868018  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.870179  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.875885  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.875947  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.879604  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.916821  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.930887  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.964114  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:46.981492  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.013576  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.013681  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.013851  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.020105  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.031162  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.036139  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.063540  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.063551  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.074041  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.087953  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.120090  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.143239  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.169705  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
I20260812 08:02:47.171011  2613 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.171195  2613 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:47.171795  2613 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137)
I20260812 08:02:47.172207  2601 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.172303  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
I20260812 08:02:47.172335  2601 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:47.172561  2135 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 1.
I20260812 08:02:47.172802  2619 raft_consensus.cc:493] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.172899  2601 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265)
I20260812 08:02:47.172933  2619 raft_consensus.cc:515] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:47.173395  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8" is_pre_election: true
I20260812 08:02:47.173447  2210 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:47.173532  2278 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 1.
I20260812 08:02:47.173839  2613 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:47.173894  2613 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.173938  2613 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:47.173892  2430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085" is_pre_election: true
I20260812 08:02:47.174034  2430 raft_consensus.cc:2468] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 1.
W20260812 08:02:47.174190  2211 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
W20260812 08:02:47.174322  2066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.174392  2065 leader_election.cc:304] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:47.174547  2601 raft_consensus.cc:2804] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 08:02:47.174626  2601 raft_consensus.cc:493] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.174678  2601 raft_consensus.cc:3060] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:47.174775  2619 leader_election.cc:290] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:47.175004  2067 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 714a507a0a7149e6a58104455bec51f2; no voters: 
I20260812 08:02:47.175289  2613 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:47.175201  2619 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260812 08:02:47.175475  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
I20260812 08:02:47.175568  2619 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d5a93c6df6741aa876e18f7cc21185c)
I20260812 08:02:47.175655  2619 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:47.175654  2613 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 election: Requested vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137)
I20260812 08:02:47.175853  2601 raft_consensus.cc:515] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:47.176326  2601 leader_election.cc:290] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: Requested vote from peers 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:47.176767  2430 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "1f20f95d7b6d4556ab254e8ed6506085"
I20260812 08:02:47.176894  2430 raft_consensus.cc:3060] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 FOLLOWER]: Advancing to term 2
I20260812 08:02:47.176966  2619 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
W20260812 08:02:47.177613  2211 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.177294  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2"
I20260812 08:02:47.177870  2278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "714a507a0a7149e6a58104455bec51f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "1d233039be434b58822feb9300ddbbd8"
I20260812 08:02:47.177989  2135 raft_consensus.cc:2393] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d233039be434b58822feb9300ddbbd8 in current term 2: Already voted for candidate 714a507a0a7149e6a58104455bec51f2 in this term.
I20260812 08:02:47.178025  2278 raft_consensus.cc:2393] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 714a507a0a7149e6a58104455bec51f2 in current term 2: Already voted for candidate 1d233039be434b58822feb9300ddbbd8 in this term.
I20260812 08:02:47.177362  2619 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: Requested vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 1d233039be434b58822feb9300ddbbd8 (127.22.208.131:39265)
I20260812 08:02:47.178395  2210 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8; no voters: 3d5a93c6df6741aa876e18f7cc21185c, 714a507a0a7149e6a58104455bec51f2
W20260812 08:02:47.178412  2066 leader_election.cc:336] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
W20260812 08:02:47.178503  2066 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.178664  2613 raft_consensus.cc:2749] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260812 08:02:47.178676  2066 leader_election.cc:336] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.179121  2430 raft_consensus.cc:2468] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 714a507a0a7149e6a58104455bec51f2 in term 2.
I20260812 08:02:47.179275  2616 raft_consensus.cc:493] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:47.179399  2616 raft_consensus.cc:515] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
I20260812 08:02:47.179665  2065 leader_election.cc:304] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085, 714a507a0a7149e6a58104455bec51f2; no voters: 3d5a93c6df6741aa876e18f7cc21185c
W20260812 08:02:47.179844  2066 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.179932  2066 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 714a507a0a7149e6a58104455bec51f2; no voters: 1d233039be434b58822feb9300ddbbd8, 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:47.179937  2619 raft_consensus.cc:2804] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 08:02:47.180089  2601 raft_consensus.cc:2749] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 08:02:47.180243  2619 raft_consensus.cc:697] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 2 LEADER]: Becoming Leader. State: Replica: 714a507a0a7149e6a58104455bec51f2, State: Running, Role: LEADER
I20260812 08:02:47.180287  2616 leader_election.cc:290] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137), 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759)
I20260812 08:02:47.180413  2619 consensus_queue.cc:237] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } }
W20260812 08:02:47.181994  2362 leader_election.cc:336] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:47.182644  1818 catalog_manager.cc:5719] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 reported cstate change: term changed from 1 to 2, leader changed from 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) to 714a507a0a7149e6a58104455bec51f2 (127.22.208.130). New cstate: current_term: 2 leader_uuid: "714a507a0a7149e6a58104455bec51f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:47.183396  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
I20260812 08:02:47.184569  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" candidate_uuid: "1f20f95d7b6d4556ab254e8ed6506085" candidate_term: 3 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
W20260812 08:02:47.184726  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
I20260812 08:02:47.185024  2361 leader_election.cc:304] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f20f95d7b6d4556ab254e8ed6506085; no voters: 3d5a93c6df6741aa876e18f7cc21185c, 714a507a0a7149e6a58104455bec51f2
I20260812 08:02:47.185273  2616 raft_consensus.cc:2749] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260812 08:02:47.197058  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.224603  2388 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37534: Illegal state: replica 1f20f95d7b6d4556ab254e8ed6506085 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.225414  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
I20260812 08:02:47.236930  2430 raft_consensus.cc:1275] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Refusing update from remote peer 714a507a0a7149e6a58104455bec51f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
W20260812 08:02:47.237075  2066 consensus_peers.cc:597] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:47.237659  2601 consensus_queue.cc:1048] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
W20260812 08:02:47.251470  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.292409  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.293766  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.298215  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.306047  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.313470  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.321872  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.337916  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.343418  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.368510  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.396458  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.411784  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.412761  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.446470  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.457753  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.465936  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.469847  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.481156  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.508427  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.519611  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.530179  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.542023  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.592737  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.608479  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.632841  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.642565  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.657202  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.666185  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.690456  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.707980  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.720912  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.729722  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.752957  2486 meta_cache.cc:302] tablet 071986b2c7af4287bb4a2e6063c28a69: replica 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759) has failed: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
W20260812 08:02:47.783073  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.788378  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.814766  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.839375  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.853482  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.861268  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.885671  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.903504  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.928933  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.957973  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:47.990177  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.038620  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.039086  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.058491  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.067320  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.081022  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.106361  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.135493  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.167193  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.197228  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.201988  2211 consensus_peers.cc:597] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:02:48.209877  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.248955  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.269007  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.277350  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.295838  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.319461  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.339879  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.348541  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.415688  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.426100  2362 consensus_peers.cc:597] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260812 08:02:48.432250  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.466940  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.483179  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.490429  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.494221  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.540092  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.569511  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.580495  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.629894  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.643991  2241 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49508: Illegal state: replica 1d233039be434b58822feb9300ddbbd8 is not leader of this config: current role FOLLOWER
I20260812 08:02:48.689875  2652 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:48.690040  2652 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:48.690560  2652 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137)
I20260812 08:02:48.691191  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2" is_pre_election: true
I20260812 08:02:48.691390  2135 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 2.
I20260812 08:02:48.691802  2210 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 714a507a0a7149e6a58104455bec51f2; no voters: 
W20260812 08:02:48.691974  2211 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:48.692013  2652 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 08:02:48.692092  2652 raft_consensus.cc:493] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:48.692142  2652 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 2 FOLLOWER]: Advancing to term 3
I20260812 08:02:48.693049  2652 raft_consensus.cc:515] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:48.693492  2652 leader_election.cc:290] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 election: Requested vote from peers 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759), 714a507a0a7149e6a58104455bec51f2 (127.22.208.130:44137)
I20260812 08:02:48.693991  2135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" candidate_uuid: "1d233039be434b58822feb9300ddbbd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "714a507a0a7149e6a58104455bec51f2"
I20260812 08:02:48.694195  2135 raft_consensus.cc:3060] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 2 FOLLOWER]: Advancing to term 3
W20260812 08:02:48.694440  2211 leader_election.cc:336] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111)
I20260812 08:02:48.695374  2135 raft_consensus.cc:2468] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d233039be434b58822feb9300ddbbd8 in term 3.
I20260812 08:02:48.695798  2210 leader_election.cc:304] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d233039be434b58822feb9300ddbbd8, 714a507a0a7149e6a58104455bec51f2; no voters: 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:48.696054  2652 raft_consensus.cc:2804] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 08:02:48.696290  2652 raft_consensus.cc:697] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 3 LEADER]: Becoming Leader. State: Replica: 1d233039be434b58822feb9300ddbbd8, State: Running, Role: LEADER
I20260812 08:02:48.696483  2652 consensus_queue.cc:237] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.34, Last appended by leader: 34, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
W20260812 08:02:48.697348  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
I20260812 08:02:48.698706  1827 catalog_manager.cc:5719] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 reported cstate change: term changed from 1 to 3, leader changed from 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) to 1d233039be434b58822feb9300ddbbd8 (127.22.208.131). New cstate: current_term: 3 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5a93c6df6741aa876e18f7cc21185c" member_type: VOTER last_known_addr { host: "127.22.208.129" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } }
W20260812 08:02:48.721834  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.721936  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:48.735582  2089 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
I20260812 08:02:48.777637  2135 raft_consensus.cc:1275] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20260812 08:02:48.778215  2667 consensus_queue.cc:1048] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20260812 08:02:48.778472  2211 consensus_peers.cc:597] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 08:02:48.802160  2092 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:43488: Illegal state: replica 714a507a0a7149e6a58104455bec51f2 is not leader of this config: current role FOLLOWER
W20260812 08:02:49.576629  2066 consensus_peers.cc:597] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 -> Peer 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): Couldn't send request to peer 3d5a93c6df6741aa876e18f7cc21185c. Status: Network error: Client connection negotiation failed: client connection to 127.22.208.129:33759: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 08:02:49.748545 23362 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskTcd653/build/debug/bin/kudu
/tmp/dist-test-taskTcd653/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.208.129:33759
--local_ip_for_outbound_sockets=127.22.208.129
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--builtin_ntp_servers=127.22.208.148:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--raft_prepare_replacement_before_eviction=true with env {}
W20260812 08:02:49.875519  2720 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:49.875787  2720 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:49.875833  2720 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:49.879503  2720 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:49.879630  2720 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.208.129
I20260812 08:02:49.883916  2720 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.208.148:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=10
--fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new
--fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.208.129:33759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/info.pb
--webserver_interface=127.22.208.129
--webserver_port=0
--tserver_master_addrs=127.22.208.190:38821
--never_fsync=true
--heap_profile_path=/tmp/kudu.2720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.208.129

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:49.885280  2720 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:49.886513  2720 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:49.894480  2726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:49.894480  2725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:49.894948  2720 server_base.cc:1061] running on GCE node
W20260812 08:02:49.894500  2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:49.895511  2720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:49.896227  2720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:49.897462  2720 hybrid_clock.cc:648] HybridClock initialized: now 1786521769897448 us; error 29 us; skew 500 ppm
I20260812 08:02:49.899614  2720 webserver.cc:533] Webserver started at http://127.22.208.129:44797/ using document root <none> and password file <none>
I20260812 08:02:49.900213  2720 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:49.900287  2720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:49.900456  2720 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:49.902268  2720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/instance:
uuid: "096fb4c61297434290d3c98ade96da6f"
format_stamp: "Formatted at 2026-08-12 08:02:49 on dist-test-slave-dhph"
I20260812 08:02:49.902803  2720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new/instance:
uuid: "096fb4c61297434290d3c98ade96da6f"
format_stamp: "Formatted at 2026-08-12 08:02:49 on dist-test-slave-dhph"
I20260812 08:02:49.906555  2720 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:49.909843  2736 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:49.911799  2720 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:49.912082  2720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new,/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new
uuid: "096fb4c61297434290d3c98ade96da6f"
format_stamp: "Formatted at 2026-08-12 08:02:49 on dist-test-slave-dhph"
I20260812 08:02:49.912423  2720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new
metadata directory: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new
1 data directories: /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:49.927234  2720 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:49.928071  2720 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:49.928313  2720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:49.929015  2720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:49.930166  2720 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:49.930222  2720 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:49.930269  2720 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:49.930285  2720 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:49.943426  2720 rpc_server.cc:307] RPC server started. Bound to: 127.22.208.129:33759
I20260812 08:02:49.943472  2862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.208.129:33759 every 8 connection(s)
I20260812 08:02:49.944499  2720 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/info.pb
I20260812 08:02:49.948498 23362 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskTcd653/build/debug/bin/kudu as pid 2720
I20260812 08:02:49.959774  2863 heartbeater.cc:344] Connected to a master server at 127.22.208.190:38821
I20260812 08:02:49.960117  2863 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:49.960731  2863 heartbeater.cc:507] Master 127.22.208.190:38821 requested a full tablet report, sending...
I20260812 08:02:49.962162  1818 ts_manager.cc:194] Registered new tserver with Master: 096fb4c61297434290d3c98ade96da6f (127.22.208.129:33759)
I20260812 08:02:49.962895  1818 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.208.129:39203
W20260812 08:02:49.985512  2811 tablet_service.cc:409] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.22.208.132:56021: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" caller_uuid: "1f20f95d7b6d4556ab254e8ed6506085" caller_term: 1 preceding_id { term: 1 index: 33 } ops { id { term: 1 index: 34 } timestamp: 7317593152229216256 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000!\227!\tR;\246\233\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000n\312>\020Q\207\020\237\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\342v-\022\3779\014m \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\001\370\322\005V\321e\2510\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\002X\324\007\200\363\212-@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000oM\361\005V\'G\357P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0008\351L\023X(\230\017`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034\244\225\005\240\3431\231p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000eA\031\024\274=\252!\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\013\004\213\004^G\224\327\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000K\231F\006\260\241\266\274\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\220\271\222\017<\270\227\007\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000O\377E\007\251+\274\021\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""758ebd215c45b9fe5781e60da24d18ba3394347b0e03236336cec8b503b5fe5fb6fbe99ba94bf84818b6538a544d2b65317f84ba55c5d172684cf88b2aa304b1d16b2ae9a59ca1838f6dcc52e30288765cd64418ee21668ba45cda5518042a33f2d24c28990c21f9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 164 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 35 } timestamp: 7317593152231714816 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\223\035M\016\230\270\017Z\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\303\335\037\n;\340\224R\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Bb\"\017\336\024\313\316 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\335\002\260\001t\227O,0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322m\205\003\036\211\200y@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000z\215\303\014Z\242\317\267P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\376t\000\007\272u/\337`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000cR\252\006Ro\244lp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000|\263\247\007\021\3263\376\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d9c3f4f6200e9f62d291a4402f6cde8c45ed19b4a26538939004fc4deea0988bc810a69d7a55231bb9e9cbedc896e645392d9aff4235f159a26de186b7e2adcc500e09c51346d97e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 167 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 36 } timestamp: 7317593152271101952 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\016\261\204\ra\020\203\255\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\276\232\"\r\034\0101q\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\302\277D\t\317\224\216\272 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\303v:\014\313\301\217:0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\213\304}\031m\347y}@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000%z\334\025\306\204\t\366P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\020\367~\005\276G\231\370`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\361\376\022\013\232\203\217\351p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000e(\253\002\203\305\017\337\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\354\222\273\030\206\237g\334\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a89579b8dad4a95820a546bc058de2c5e55768681b0f2dfeb551b4d18e6c830749e86710f18ce2b0da0acca1f601186bd59f4b511a137bdc8a45327c89eb21f8f44dd893c58bad181eee927101a09129" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152228352000 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 173 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 37 } timestamp: 7317593152338837504 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\\\343\027\005Pd\223\004\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000]\350\031\027\214\270\320.\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\'\334\246\025,/i\253 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\350%\257\n\343\005\321n0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000f\210\355\020\376\211\002(@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\240\210\335\030\207-\207RP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\250\032\226\0319\276&\344`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c77ad230a0fae3f3f3bef2b07a5e36b433999714d11bbaadec38c8f608234a51fe9f5e0421aaf1756da3bbe431091a4958323a2a13114f9e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152271101952 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 179 first_incomplete_seq_no: 162 attempt_no: 0 } } ops { id { term: 1 index: 38 } timestamp: 7317593158822387712 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000]\324q\001\021\2434|\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\352h\223\014\245A\316\221\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000GY\337\001E/\026o \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000d\316\233\000\265r\034\2470\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\247\030\267\004\224\322,x@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\312&\345\003\353q\342\242P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a585e94af52e5ab0982861c90657bf06fe716a0d8fd378ab7e1c0126c11f4cf110b740f6de13ffc08a02968fdfd5587d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593158782095360 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 182 first_incomplete_seq_no: 165 attempt_no: 0 } } ops { id { term: 1 index: 39 } timestamp: 7317593164996513792 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\210\241\014\014\343\237\302=\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000C\330z\004\215\237=\205\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\216\320\030\010\366(a\366 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\271@\201\024\311Q\260\3030\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\017-\211\031\332\263>\274@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000U\027l\021\254\344c7P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\346\201\330\005m>I\205`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000z<K\024\212\361.Wp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000=\261,\020\277TYU\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\276\\m\006B\310\243\273\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\021\356\265\023\000\263\002\333\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\233\256\320\030\017\034\331\314\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\377]\330\003Q\276\247\247\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000N\020\334\022\022\030\214 \320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4e29d241b755a2fc4adbcf8fca2fb75cd0fac8d4012dbe03d138ee33613cbe34b5f7afa3b05865addaf53782b21eccea88f43ba734f0ab11655f37de1c99c5faf4980886d2afa26ea7c96239ac9f82d5fc0895e5af525a023bb248ff6bacf7ee817aed605aec74c492555c94c462458d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164907425792 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 187 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 1 index: 40 } timestamp: 7317593165031444480 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000r\t\310\001\302\264\374\204\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\204\233\n\002a\r\325\023\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\331G)\014\341P#\356 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\031\357\200\007pmo60\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242\356\335\024\375\004)\213@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227\233t\0019\262qJP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\016j\314\t\264\366t\022`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\264A\t\016\232H\352\375p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\"B\214\016w\327[c\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\306\265P\016\246\007\220\230\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\266\232w\nuY*)\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\264\210\025\025@t|C\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\221E\237\005\022\251\246\230\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""16d707027e1a40095925b2a8f3e8c2a235126ed79a56429a003d37bd662cb2d464da69427e68fde9766d4026fd1d770fee73452b238443c03fa6a1dca99967fa396949a15ec20cf2c4a8f62be1734888fb2d750acabe4567f87038f69f57b880f7beaa880d5bc69f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164974698496 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 193 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 1 index: 41 } timestamp: 7317593165091057664 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\250\216=\010\210\312XX\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\177F\314\022\365C\202A\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\370?\337\022\333]\302F \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\271\247\026\002\332\243Q\2270\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000M\325\343\021L\253\207\026@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\245\017\277\022\201<q\312P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000r\326\231\024e^\373\233`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\005\347S\023\216\330\203Tp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000iy+\004\240\177\301T\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000X`\033\014\367\275|@\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000_r\344\007\304j\'t\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\213\347d\004\255\244\310\254\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000E\036\307\005\274E8V\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\361\245\236\020\275r\033\203\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\305\006R\024X\035\244U\340\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""af6cbc6197877adb4dc325a68db9ba59143479f602661fae025e3c37f186c1d6b3831293c4715848600d815cad4fe090039180030ad68d93817b2c65fb3f78f16dfcb80e552f6d70deb728c295aa1245f35e30eed9bd28dc1e4a23acaad84321fcac17d50ca18023ef9e3bcb7acf49b2fce4a0f08c04593e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 195 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 42 } timestamp: 7317593165094621184 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\233\357\260\030@>d0\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322\273`\001mkP\025\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\317\337\307\020\226S\014B \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\327\223\327\022B\234X\0360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\013\374\312\014\035\215\022k@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000K@9\010\265\343\321\265P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\214#?\0057\33366`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\216Y\306\005&\360z\035p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242L\270\007\230\240}\341\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\210\364\032\003\027\225\337\361\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\310\263w\026: \220\021\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""aa2b4ff4031c639a747bc6a7545e31b8604f60d776bda7a6208698af492433afea6178214a2ba887d256b6b275e8d30b1735366dcd10b5ab9f64b27397ff99e5d050682e9dcf9a67fdbb8cc3482cebbde16371d68f8d951b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 197 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 43 } timestamp: 7317593165107884032 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000D\003P\003\3647!i\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000`sx\014\212\245\303\t\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000E\317>\n\343\233H} \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000T\231q\027\366\317\373=0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000p\355\361\027;R\301\244@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\2425\363\r@\361]\263P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\273(/\t5\372&\022`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\3556\257\016\255\341p\301p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\336{x\003\201\306\205\374\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000 \322\351\020\031\231\263\016\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5ece11a055ce5e59040182777e38a95db250ce52ae334840d37bdd0fd1dc5d356c63bafa0f1113b56b2d2a0f4448a9dc73096758dcbffad1d6cfa477346d18a80c901da01311f3a34a4bec5afd145da2" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 204 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 44 } timestamp: 7317593165171286016 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\374\315x\014\262\373\260\216\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\002\"\022\020\331\241\217L\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\274\367\177\024V\036s\244 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\0352\006\007(GZ\2710\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\346\366\353\005\235\362\036\316@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000v7\321\030;\242m\003P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000g\247E\005\305uK\237`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\023\264!\016@\261D*p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\217\253[\025)\034\204\274\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\035LN\023\233R\274\334\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0005\336\"\027\236F\204A\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bb2efe581478c38681ad4f743276931092f7380048fb2374ba8ba9724e4332ca5bcf4aeb8c7f72c8ef5e4458abb24d0e6ac227dfe18fb07505f158dbd0016b79047dcc6dbdf0567860e2d0e6160179d04b1c3b71d5053c7a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165105995776 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 209 first_incomplete_seq_no: 195 attempt_no: 0 } } ops { id { term: 1 index: 45 } timestamp: 7317593165270949888 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\326B\303\020\374\335\016\306\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000B::\016P\343\016k\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\202\035\235\rU\2728| \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\304\324h\026W\310ME0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000]2\311\027\032.\245K@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000~\017\216\006\376\363\322SP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000`\265\030\017N\332\226o`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\364\0246\0038$\r\213p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\234<\301\0207\231A\372\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034O7\002\274p\260\177\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1614978cc3d328f57eb38e48fc84b3e8d13a627b202f04264f0b8d3f7ecc55a0f08e90b03f7a576f1ae88cc4b2e38cf9d32d6727939eb835d4ebf6271edb0a58044341f3373d7569291acd073d15374c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 218 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 1 index: 46 } timestamp: 7317593165286989824 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\306\254r\006\257\242v[\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000ht\353\004_\320\235\311\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\236\266\224\022\035\276v\273 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\341\253\233\023Epnq0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3aee5ac56d12b3791d2739ebf497462892f5f47c230c053b4cfd084af056b385" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 225 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 1 index: 47 } timestamp: 7317593165332488192 op_type: WRITE_OP write_request { tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\342\316R\007n\312\256\235\000\000\000\000\000\000\000\000\020\000\000\000\000\000\00
I20260812 08:02:49.991477  2362 consensus_queue.cc:579] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: The tablet replica hosted on peer 3d5a93c6df6741aa876e18f7cc21185c has failed
I20260812 08:02:49.995244  2431 consensus_queue.cc:237] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:49.996625  2278 raft_consensus.cc:1275] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Refusing update from remote peer 1f20f95d7b6d4556ab254e8ed6506085: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20260812 08:02:49.997225  2711 consensus_queue.cc:1048] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260812 08:02:49.999545  2640 raft_consensus.cc:2955] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } }
I20260812 08:02:50.000011  2278 raft_consensus.cc:2955] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } }
I20260812 08:02:50.001812  1812 catalog_manager.cc:5232] ChangeConfig:REMOVE_PEER RPC for tablet 7f94f84d271246aa9aaef39ef0c7eaac with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 08:02:50.002924  1818 catalog_manager.cc:5719] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 reported cstate change: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New cstate: current_term: 1 leader_uuid: "1f20f95d7b6d4556ab254e8ed6506085" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: HEALTHY } } }
I20260812 08:02:50.007562  2431 consensus_queue.cc:237] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } }
I20260812 08:02:50.009276  2278 raft_consensus.cc:1275] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Refusing update from remote peer 1f20f95d7b6d4556ab254e8ed6506085: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
W20260812 08:02:50.009404  2362 consensus_peers.cc:597] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 -> Peer 096fb4c61297434290d3c98ade96da6f (127.22.208.129:33759): Couldn't send request to peer 096fb4c61297434290d3c98ade96da6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f94f84d271246aa9aaef39ef0c7eaac. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:50.009786  2711 consensus_queue.cc:1048] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
W20260812 08:02:50.011132  2786 tablet_service.cc:409] Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.0.0.1:39020: tablet_id: "7f94f84d271246aa9aaef39ef0c7eaac" reason: "TS 3d5a93c6df6741aa876e18f7cc21185c not found in new config with opid_index 129" delete_type: TABLET_DATA_TOMBSTONED dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" cas_config_opid_index_less_or_equal: -1
I20260812 08:02:50.012259  2640 raft_consensus.cc:2955] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
W20260812 08:02:50.012825  1813 catalog_manager.cc:5036] TS 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): delete failed for tablet 7f94f84d271246aa9aaef39ef0c7eaac because the server uuid is wrong. No further retry: Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:50.012938  2278 raft_consensus.cc:2955] T 7f94f84d271246aa9aaef39ef0c7eaac P 1d233039be434b58822feb9300ddbbd8 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.015019  1812 catalog_manager.cc:5232] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7f94f84d271246aa9aaef39ef0c7eaac with cas_config_opid_index 129: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260812 08:02:50.015872  1818 catalog_manager.cc:5719] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085 reported cstate change: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New cstate: current_term: 1 leader_uuid: "1f20f95d7b6d4556ab254e8ed6506085" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:50.042838  2811 tablet_service.cc:409] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.22.208.130:46361: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" caller_uuid: "714a507a0a7149e6a58104455bec51f2" caller_term: 2 preceding_id { term: 1 index: 33 } ops { id { term: 2 index: 34 } timestamp: 7317593158372470784 op_type: NO_OP noop_request { } } ops { id { term: 2 index: 35 } timestamp: 7317593158594449408 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000:\177\223I\373}\3272\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\305\240\235@\327\324\376\307\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\005\036\0209\256\351\250\316 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000(\351\'GJ\2431\3700\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000<[D8\0253\224\000@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\316}\377Hk7\324.P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\t\243dD\231\2212\222`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\312\221_E\034\021\374\375p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\226\022\3505Z\266*D\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1af53f5d362600fd3428b71a3a1a4d6dc9e11651b1008b778ab26db49080d578515c2b98f46463c519f8a882a7fe39e7ef71411468cf40c2aa136877c05f261278dcbe28c890e42b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152271101952 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 176 first_incomplete_seq_no: 163 attempt_no: 50 } } ops { id { term: 2 index: 36 } timestamp: 7317593158618230784 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000a\366\023L\205\276\2643\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\335\317\3614\261\227q:\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\223%\\A=\226\275\211 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\264\007\231?\214\232\244C0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000,\206\275C?X\202\'@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000P\237\377;9y\201\032P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\230\223y8\317\031b\236`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000$>\n>Y\305\371sp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5d264f81fc5b9bdb1379f5b557e4902043e520d46f9c8096f9737fba9b8d954239c7e239e3aa954a993e4a3d90f83122ed9e278cd46d211a6e402338e990d428" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152228352000 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 174 first_incomplete_seq_no: 163 attempt_no: 50 } } ops { id { term: 2 index: 37 } timestamp: 7317593158782095360 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\r\002\267F\006\033\353\030\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\244W\007C|\303\255\317\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000!\263\3349\025\212\214N \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\3709\367G/\034\247\3010\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000}^\2274\3066Q!@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000]\316\003>D\234<\230P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000:\206[5\367>\023S`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Y\010RG\321\244\345\315p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000FX\310>\313\024a\261\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000[\303z=\244\304\374\266\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0002\356\036A\213lr\026\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""59df7248b99006214a8119afc37aacbf88bce4155244465c995079b1e4cb028cf8e1ab2ecf5b2e04783f54050b4a62399764d76a3af4f84bdebe4b94bea8244889a5ac6037afaf25986af371ab295a213ebac607b03e6d22" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 163 first_incomplete_seq_no: 163 attempt_no: 51 } } ops { id { term: 2 index: 38 } timestamp: 7317593158822273024 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\247d\253<H\216\274\215\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000h\254\366B\263\306\006\025\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\033E\344<\274\236!\007 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\017\276#L\2628C!0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\"}\0328\305\266\026\023@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0002\036\020H\371`JeP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\266:\3559\367\033dC`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000-4p3)n\325\314p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0003z\3518\326\007\232\327\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0c5e689114461bded065d32e1a16f2d6635a95289fed501b0a26632e3c72cc6d9c7dbd88864cb651313b43c353bba0ea7c75cbb633227179f021a249e04b4044eb18dda65affa6c9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593158782095360 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 183 first_incomplete_seq_no: 165 attempt_no: 0 } } ops { id { term: 2 index: 39 } timestamp: 7317593159119810560 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000X&\307BW\261\203\022\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\\S~J\210\311\203\270\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\231\205 9i\034Lk \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\225a|?\246l\033]0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3aa88de21ff61d08c76c4c4a2e0924290f7cd08716bcae050e39134a0a42cd23" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 165 first_incomplete_seq_no: 165 attempt_no: 53 } } ops { id { term: 2 index: 40 } timestamp: 7317593164957786112 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000w\226 7\264\362~\206\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\323+~E\016$;\335\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\357]S3\300\346M1 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\342\315^=A\226\343\0030\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\326r\345C\260\350\327\316@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000K\025\215@\335\265\323@P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\317(\037=6\202\250.`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""81e90083504132dfdd67c5bf0fcf1b5cbdccfe44ba92c5c43212112100454e6b58bc7006f6bd8edaaee39ecb439c72f3ecb4fc293c2dfd5a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164907425792 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 186 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 2 index: 41 } timestamp: 7317593165013467136 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000[\005\343<\354\360\002\363\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\211\262\337@(\247\030\230\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\315\272\0008X\037\276a \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\353\377\214>\301zd{0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000j\212\035F\304\\\000\021@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000S\003\022?\265\276ctP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Fm\241:q\t\364\004`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\353#\361K\330L\225`p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000j9>C\366b\n\216\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\347\315\300Kn\0179\230\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000,\326lLX\033\321m\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9261911506f8064101d7eaee8d01a20fabc06650c8fa8ae8557c2eec17d925f272381e458496fd31fadc674b3803163bf47ea46b4cb90f73849d4b84e591d49a56d7d8e4e9c11a8dc22d0f84b8ab8a1a16074eb77381b75a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164974698496 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 191 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 2 index: 42 } timestamp: 7317593165084921856 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\337\007\303<2\236\270#\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000NF\361F\342\354\271\237\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\231l(D\224\006\2063 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\366\375+C\230\357\206\3740\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\312%\013J\236\250\247{@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\210\352\337<\031H+)P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000S\267\270G\311\235EI`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5fb803a8d3f0776035c0553fcc39aefc92945af993c17794181db0540a46ce5524b0b9881cc94513ef6514722a42c15aa2c0cb063ea2117a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 196 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 2 index: 43 } timestamp: 7317593165101187072 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000#\371\2579\334\003\364\247\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\327\371\332?#(\340\213\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000(\000\261D:M#\357 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\211\223Y7\227\360\251\3420\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\312\342\374I`\265\n1@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\317\022\303C+\273\006\237P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\023j\362?\3165\250\350`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\306\247\2375\374\201\341\315p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\0013);ph\365\301\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\223\367\2763o\331\216\246\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\207J/L\215*G6\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""72606c79b0e36d32daf7ce1905f7b2486e6aa1b9b53f62a20464809a446e016d0fd03cdee0987783707cb4a99671f9e2d008cbd0e2f9db79add27faead7961ac43fee19425177b3170cd1da0d4a85a7610d25f0446b81079" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 201 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 2 index: 44 } timestamp: 7317593165135491072 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000R\312\202B\r \254\323\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Rio?q\364\277Y\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\335\226\212>i\027\236; \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\324\203e8gSi^0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\005\252\3367\007\311\026\327@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\030\214E\246\350/\333P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000w\341\242H\216\263\251\231`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000i?\017Etl*<p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000I\354\245HD\211\327D\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000U\351\256;\031\254n\265\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""751656459af4102264545f32be6bfe27c4ebc10ba39db90da08f4da4c064f7058f076c49e0acaa4f6efed67b39667aaffdb0e7a0526db20858b230b23536059155f3e51cfe8d7ab32a0d785734af5aa5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165036326912 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 205 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 2 index: 45 } timestamp: 7317593165176885248 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000H\215\027F\027\222\226\365\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000A\252sC\351Z\253\237\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\371\361l8\310C\364\252 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\212\222\253F\237\361_\2170\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\236\377$<?\271\273\226@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\343e@L6\004\177\305P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\271\374FD\337\371\2667`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\350$\347D1\205\366\236p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000[\337\342D\234\211P/\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""37452ee5be13f31f1aece5e3edd1916ffb12c5f49b04aa98d1801b2fe2f1a8ecde2c96e402ccc46cd0e7b1dc369bd65f1fcdefde127dd04475a490178de024b00c5012bfb9fdc17a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165105995776 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 213 first_incomplete_seq_no: 195 attempt_no: 0 } } ops { id { term: 2 index: 46 } timestamp: 7317593165278846976 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000U\214f7\205\350\254\211\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0000\201\244@\250j\203\210\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\027\305r4\203\336\203\236 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\321\256n9\266H\306\3410\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000{GMJ$-\212\357@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0005gG9\214\365p\267P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000K\r\256I\341\354\300=`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\'\217\352C\334\376\357zp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\200\02553\253\204\242\027\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\027f\213A?\314\026\270\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bc746ae77e2d0565a955b03e714b974a9b2dc5fb3441df5b16920e7a1de8ee53ad094d747b78238ae9a5e4ef94b27a327a3fb9345b3045983b54f79e0e99890145581035312014a1babf9a6af265c836" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 224 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 2 index: 47 } timestamp: 7317593165322702848 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\007\372\312F\313\321\344\034\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\335C{<\"?\303\246\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\353\345\332Em*\266\206 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\311H\374E\246\332\346Y0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\374yzJ\306q1\327@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0008\356-KR\341\322\230P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\276-\222:\343\360\265\367`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000P\031\n=w`%qp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\007\263\342AR\273~\003\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\267\334\352A1,<\277\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\367\226\2008\230n-?\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\302\334bA\247q5\231\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\203T\0049b5;\205\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\343L[? CvR\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""b46e2649d7277cad39b8128f47950fc71ba2ec1fca782c722799a0dbae1f7e003be3564c946dae3e2809dad43dfb51aa40685fc288cdb52ecf336856f00e1d72dbe04e8cc689f0e75740a00ca7974129288894873d9e7fb6d3fc05760d952074920800e916e7947dca119e9a95876ce7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 221 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 2 index: 48 } timestamp: 7317593165346902016 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\230\010-=\241\213s\002\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\014s34\273\322e?\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\261T);\216\003\327\362 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000@\341L?\026\033<\3260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\327\330/H\276\244d\211@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\377\227><r\204s\354P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0009\237\3377=\032\016v`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\030\360\321F(\272\211\352p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\251\356\324J\037\364\356\355\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a2a8bc29e7b2b4b250a14ae81a799bbf5d83ccbdefed7883ae3a29a5d9089058de9138061fea432fd528bb794a581f9243321a7c1cd1890fc436d6565863548649aea35cd0ce259d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 233 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 2 index: 49 } timestamp: 7317593165351702528 op_type: WRITE_OP write_request { tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" schema { columns { name: "key" type: INT32 is_
I20260812 08:02:50.046644  2066 consensus_queue.cc:579] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: The tablet replica hosted on peer 3d5a93c6df6741aa876e18f7cc21185c has failed
I20260812 08:02:50.050503  2135 consensus_queue.cc:237] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.129, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } }
I20260812 08:02:50.052071  2431 raft_consensus.cc:1275] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Refusing update from remote peer 714a507a0a7149e6a58104455bec51f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 129. Preceding OpId from leader: term: 2 index: 130. (index mismatch)
I20260812 08:02:50.052757  2633 consensus_queue.cc:1048] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260812 08:02:50.055509  2685 raft_consensus.cc:2955] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 2 LEADER]: Committing config change with OpId 2.130: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } }
I20260812 08:02:50.055987  2431 raft_consensus.cc:2955] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Committing config change with OpId 2.130: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } }
I20260812 08:02:50.058531  1812 catalog_manager.cc:5232] ChangeConfig:REMOVE_PEER RPC for tablet b5224a5c2c4a471e9d3be65b16da7189 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 08:02:50.059006  1818 catalog_manager.cc:5719] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 reported cstate change: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New cstate: current_term: 2 leader_uuid: "714a507a0a7149e6a58104455bec51f2" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: HEALTHY } } }
W20260812 08:02:50.062045  2786 tablet_service.cc:409] Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.0.0.1:39020: tablet_id: "b5224a5c2c4a471e9d3be65b16da7189" reason: "TS 3d5a93c6df6741aa876e18f7cc21185c not found in new config with opid_index 130" delete_type: TABLET_DATA_TOMBSTONED dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" cas_config_opid_index_less_or_equal: -1
W20260812 08:02:50.062489  1813 catalog_manager.cc:5036] TS 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): delete failed for tablet b5224a5c2c4a471e9d3be65b16da7189 because the server uuid is wrong. No further retry: Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:50.062852  2135 consensus_queue.cc:237] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 2.130, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } }
W20260812 08:02:50.064601  2066 consensus_peers.cc:597] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 -> Peer 096fb4c61297434290d3c98ade96da6f (127.22.208.129:33759): Couldn't send request to peer 096fb4c61297434290d3c98ade96da6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5224a5c2c4a471e9d3be65b16da7189. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:50.064633  2431 raft_consensus.cc:1275] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Refusing update from remote peer 714a507a0a7149e6a58104455bec51f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20260812 08:02:50.065124  2633 consensus_queue.cc:1048] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20260812 08:02:50.067023  2685 raft_consensus.cc:2955] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 [term 2 LEADER]: Committing config change with OpId 2.131: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.067517  2431 raft_consensus.cc:2955] T b5224a5c2c4a471e9d3be65b16da7189 P 1f20f95d7b6d4556ab254e8ed6506085 [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.068718  1812 catalog_manager.cc:5232] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b5224a5c2c4a471e9d3be65b16da7189 with cas_config_opid_index 130: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260812 08:02:50.069751  1818 catalog_manager.cc:5719] T b5224a5c2c4a471e9d3be65b16da7189 P 714a507a0a7149e6a58104455bec51f2 reported cstate change: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New cstate: current_term: 2 leader_uuid: "714a507a0a7149e6a58104455bec51f2" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f20f95d7b6d4556ab254e8ed6506085" member_type: VOTER last_known_addr { host: "127.22.208.132" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 08:02:50.128799  2811 tablet_service.cc:409] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.22.208.131:45579: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" caller_uuid: "1d233039be434b58822feb9300ddbbd8" caller_term: 1 preceding_id { term: 1 index: 32 } ops { id { term: 1 index: 33 } timestamp: 7317593152169889792 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\307|\336\033\275\001\275f\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000-\014\0230\336\177\241\226\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000|\374\263!\272\260\273\325 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\032\267\353\035:\316\016F0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\036C]/]\351O\271@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\376\326\024\033M\3377\323P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000F\002\225*\352\344\350\244`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\276\302~\037\244\357l\215p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\212\003+&\330\332\254\307\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\314\373\211%\177Vj\333\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\376\352\177$)\331\312,\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\010\275\022$\031|\346=\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000^\230\212.\256\364\261\373\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000i\na%\231Gg\365\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\374\267:\'wK\307\315\340\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""edea84397f1da37795c7de33e99a2488913f3b3b9c939052ae2e31b23581699bbd30d5db2250787fc2c0769153505600c1c9b3a2c260e4d8f35a6bd3eaba499ecea643590b1f1f25b8723f1fa91c94005f2a06408001091fa99173721864ad601ed8046f70164675f2e53c5bcfbb6a9677b60bbc2179006e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152065200128 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 155 first_incomplete_seq_no: 153 attempt_no: 0 } } ops { id { term: 1 index: 34 } timestamp: 7317593152246411264 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000R\226\222-\013\260\230\353\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\321*\317 \222w\301\261\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\222+*2~S7Z \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\330\250c(\252\361\362\3030\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\306\234\263\031\036\352\200T@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\263A\200\035\274\317#tP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\304\301\311\035\303\344\230\376`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\200p\013\037\373KW`p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\177\223i(\035vI\270\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000rw\025(\231\013\327\227\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242n\003$\250\216tN\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""64fcb19713de86f3e3680ce0884af2391719070c583b615917c10e5a872ac771314c5ef919de8d70b49b01a6d912ea31a6b6440a905a42695cf1c68d0a44c3a75ac813668ae37834778dae7eed761f0787671e7e0605cbfc" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 162 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 35 } timestamp: 7317593152249647104 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\313)\2021\276w6\277\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\211\362\274\037\315\031\206\002\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000x5.\037D|\007\t \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\343\320\367$&\355=\3050\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Yf)\"\206\335\r-@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000b\326\324%P\362\325\260P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\014;\223&\227E\026\315`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\261\r\007(\220\335\367Np\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\314\342\033\261\335\217\325\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\000r\'&\304\033\373\253\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\200\324\017&\016\273\r\223\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6d463080b634b71811e4dc499074cd402a1db81c0a4497bceef33eeac74f5b1c111785623e6bdb6443cddbe4003eaf5f220cc34ae7d2c3206026ad829692c63ec8f07ea24a13848723a84587f8d9aa5c2641efa565b89a84" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 169 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 36 } timestamp: 7317593152259960832 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\371c\304/>\006\367\236\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\204\247\322 \201\337\354\354\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\007\244\226#Ue\202\264 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\302\007\260/\334\022\353\3500\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Z\030\307\034\342]\3671@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000@\257\335)\020Ch\250P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Dp72\333\021\023\365`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\224\270g0\031F\214\243p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\027\367\223&\256\342\223Y\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1ceed033054da63894e5c647f27915b842d66aef960d26624ca3ca65aa79a8641b5191089011a61d5f04b5aedc67c67c5abbe098c92c936376415797f7bb6e501f681d5158b981e5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152167272448 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 170 first_incomplete_seq_no: 155 attempt_no: 0 } } ops { id { term: 1 index: 37 } timestamp: 7317593152358993920 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\3464N%\020\037\310s\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\n\030]\"\342\013\367-\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\245\337!.)^\216d \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000{\341m#\377*\002\3610\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000.*\332\"^\273\371c@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000a\343N\"mPa\323P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\'\217\317\033m\027\027S`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\316\340\034\036-PH\264p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\025\233\200&\345D\323Z\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000l\214\266#\336@\245\241\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\261~\264+[\024U\207\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\033o\024 =\3652\351\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000t\250\207\'D\373\n\t\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000/\215\006\036\254\241d\271\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2e84d4eb256d5052e99d99b853ae28c01f6e0358df3b6eb42867d48c7166901aa0239d3c044fbd40d0758fe12c720a70320bd9b45003a42efb5e1badbed67443e8f0149dd3b30a7b1da2aa7044c75c86315ed7ea1362b9f139ae905be70f098ca6263a792f91e300e0d8e6e6803e6f6d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152271101952 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 177 first_incomplete_seq_no: 162 attempt_no: 0 } } ops { id { term: 1 index: 38 } timestamp: 7317593158840029184 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\006:\217&\035l\021\246\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0002\240\361$\023,k\024\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000{*\271/\247\340\t\232 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000C\322q2\341\306nA0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000ie])\267)\035u@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000(\373\014(2(\324\035P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\326\031\n T\261wp`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\354]\212)_\265H=p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Jqh,\343\313\260Q\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0003\006\"\036\213\023\010\366\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Gl\217\036;\030\354\272\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\260\312\345\037\345\313\221\n\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000^\350\261\033\273\243Y\003\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\251t%\227*5#\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\307z\023\"\362\"\375\224\340\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000s^\007 \"\177\256z\360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000}7\\(_\242/\033\000\001\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000d\317s\036\265V\234\t\020\001\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3f129ebc3ee6d1fbc5b92efd0177cdf5bf9c91e6005d67554584c801a0cdba9edd24b196a80e3d4071c6ce43e61f9620c4ef1512d8a0e8322001f67141b50a43101118b124d8296903e03913eb86626a37692736311acd4b5f225f58134334c0429f2875be3276fd5fc77049bd39f37efaa3bf8828e72f973ed143903955865088cf6af1d2ec331e22322184926d26f3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593158782095360 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 184 first_incomplete_seq_no: 165 attempt_no: 0 } } ops { id { term: 1 index: 39 } timestamp: 7317593164962275328 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\017\267\n0\nzk\244\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000B\235\333#xvC\035\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\247%\2012\030\364Z> \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000z\250\237\033\217o\027\2470\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\207.~2\035\253\322z@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000]?I-\311\253\002\210P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000S`}&\203\223L^`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0000\261n,\334\341Q\014p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0008(\"!\034Fp\021\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""65fd45902dc408d8f0c1900567aca28971987af0a35eb9a76982635fde94502d62bec73ab613c6f7a8417fecc86b28da15b5f91b3beacd7f1ae42974f2bb5a941c23c86e62777861" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164907425792 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 188 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 1 index: 40 } timestamp: 7317593165043142656 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\245s~&\354{<\344\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\266\371@\033\273\014x\002\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0008\340+/\3223\037\316 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\177V\315&\211\317+\3200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000@\312\336\0369j\036\237@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\324\203\304+\302\363\3272P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000E\0326/n$\306\177`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000|\017\372 \354\036\351hp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\200\335_*\371l\001\313\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\361H03K\372,1\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000g\352\024,\210e\262\244\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\247\034\201 \237Y\352A\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7abdb9f87bf8cdd12d45bebd7bdb0306cc7801a8356985fabeaa8909dc15ddadc7b8660f3668cd7ddab14247d6ece3aa8945eb1b9a9cbe645dc6270c785b2832d085db7a08185e1fb69b45dd980fda01adbb0e702761e50904c10105b654250d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164962275328 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 190 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 1 index: 41 } timestamp: 7317593165159145472 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\321\263\206\035z=\265\010\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\007\247k+\277\340\211\354\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\211^\277\032e\376:l \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000}E(!X\352\266\2620\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\301c]\035\321t\267D@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000x\\4-\241KK\246P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\366|q/\220f\333.`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""23d6df475fb98165d40788e3f558345c23eb71acece80ce76c2263f48ba0bf45c723eff810bf0ce30bc22f54f362b54eca9d3cf7b21b44cc" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 199 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 42 } timestamp: 7317593165163479040 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000Jp\307\"&\235\240o\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000%p=&\262\030\0336\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\323Y\212&\021\371\223\022 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\255\224=2\211\361\341\3360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000(\221B\033\245\250\223\231@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000DJv%P\316\364jP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""e320ba2c733d6170af82426fa1af7d35e531750aa3f62eda04373bcd8c6e6570eac1348764bf41fd9612fad1f021b6f9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 202 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 43 } timestamp: 7317593165167468544 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\343\3503\036*\327`\341\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000r\340\3231\215\000*/\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Fc=%\356\226Jb \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\353\236\303#\353\200\371\3510\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000F\253\313$\251\303\216R@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\316\0130\341}\211\203P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9bf3c728433042dc10eb5a4b96ea36dc45d84a41f5e62a737ea61522136b16a0db183651252c1b79c541c189c01259c9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165036326912 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 206 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 1 index: 44 } timestamp: 7317593165207244800 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\203J\361)\\/\017&\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\251\326\250\034gq\237@\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000B\316\2321\222\177)\327 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\222\342|&Hf\002\2220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\243\255\265\031\274!\0037@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000u\215\377.\270G\235:P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\021\355\231\036\022|\332\356`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\020\215\310!\214\010\224\255p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\3107\244\'\017\263\242\254\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\366.p$\312\363\256^\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\317\3528)\\\016>\362\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""10780338ee36f59bdd32973bc8cda494623d9ef4771d6d77c0fe856f8bd4b77e7cfde8620347326d25eafb904c537cc520b5dc638730325f2ea946ac56bdbc450312810967aaeee41e9fcdde1b09730e75cda961225675bf" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165105995776 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 214 first_incomplete_seq_no: 195 attempt_no: 0 } } ops { id { term: 1 index: 45 } timestamp: 7317593165270937600 op_type: WRITE_OP write_request { tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000n\r\\.{\251\300\237\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0001tp\036L\230\302\251\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\025TK&E\324J* \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227\265\336.\241\355\313\3370\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\325qF.\362v\221\321@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000qf\344\'n\241\030<P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000&D. vh\376\242`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\027\335\262-^0\266\322p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322\263\2421#\356L\200\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000V\343T1\252-\\1\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\377\222\247\'\252[u\177\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322|\203#{\365G\265\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\335\332\234#\014R\255\331\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\037\241\375\036\224\030\322\221\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""75e2525a3b195cb27ddc67932228073f41bd260b52193a22d20080a9bdfe029c2bf540b964d8907500cf46cb506173ae2634b8cc1ea0d959fef7049c169f7c1fa63a4054f4aafe7566d2450f6d67df6404216c165c7e34f093e78f1e0eca098f70b0a44cb9e566c1ce3812a786a1741f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017
I20260812 08:02:50.132176  2211 consensus_queue.cc:579] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: The tablet replica hosted on peer 3d5a93c6df6741aa876e18f7cc21185c has failed
I20260812 08:02:50.135675  2278 consensus_queue.cc:237] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:50.137138  2135 raft_consensus.cc:1275] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20260812 08:02:50.137789  2695 consensus_queue.cc:1048] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260812 08:02:50.140408  2652 raft_consensus.cc:2955] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
I20260812 08:02:50.141047  2135 raft_consensus.cc:2955] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
I20260812 08:02:50.142273  1814 catalog_manager.cc:5232] ChangeConfig:REMOVE_PEER RPC for tablet 3ecf598b966e4dcabcd62a0ba48180aa with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 08:02:50.143476  1818 catalog_manager.cc:5719] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 reported cstate change: config changed from index -1 to 129, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New cstate: current_term: 1 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
W20260812 08:02:50.146189  2786 tablet_service.cc:409] Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.0.0.1:39020: tablet_id: "3ecf598b966e4dcabcd62a0ba48180aa" reason: "TS 3d5a93c6df6741aa876e18f7cc21185c not found in new config with opid_index 129" delete_type: TABLET_DATA_TOMBSTONED dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" cas_config_opid_index_less_or_equal: -1
W20260812 08:02:50.146823  1813 catalog_manager.cc:5036] TS 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): delete failed for tablet 3ecf598b966e4dcabcd62a0ba48180aa because the server uuid is wrong. No further retry: Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:50.147068  2278 consensus_queue.cc:237] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } }
W20260812 08:02:50.148694  2211 consensus_peers.cc:597] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 -> Peer 096fb4c61297434290d3c98ade96da6f (127.22.208.129:33759): Couldn't send request to peer 096fb4c61297434290d3c98ade96da6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ecf598b966e4dcabcd62a0ba48180aa. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:50.148677  2135 raft_consensus.cc:1275] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20260812 08:02:50.149082  2652 consensus_queue.cc:1048] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260812 08:02:50.151072  2695 raft_consensus.cc:2955] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.151520  2135 raft_consensus.cc:2955] T 3ecf598b966e4dcabcd62a0ba48180aa P 714a507a0a7149e6a58104455bec51f2 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.152897  1814 catalog_manager.cc:5232] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ecf598b966e4dcabcd62a0ba48180aa with cas_config_opid_index 129: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260812 08:02:50.153817  1827 catalog_manager.cc:5719] T 3ecf598b966e4dcabcd62a0ba48180aa P 1d233039be434b58822feb9300ddbbd8 reported cstate change: config changed from index 129 to 130, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New cstate: current_term: 1 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:50.400517  2876 ts_tablet_manager.cc:933] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: Initiating tablet copy from peer 1f20f95d7b6d4556ab254e8ed6506085 (127.22.208.132:46427)
I20260812 08:02:50.401155  2876 tablet_copy_client.cc:323] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: tablet copy: Beginning tablet copy session from remote peer at address 127.22.208.132:46427
I20260812 08:02:50.405434  2451 tablet_copy_service.cc:140] P 1f20f95d7b6d4556ab254e8ed6506085: Received BeginTabletCopySession request for tablet 7f94f84d271246aa9aaef39ef0c7eaac from peer 096fb4c61297434290d3c98ade96da6f ({username='slave'} at 127.22.208.129:41297)
I20260812 08:02:50.405687  2451 tablet_copy_service.cc:161] P 1f20f95d7b6d4556ab254e8ed6506085: Beginning new tablet copy session on tablet 7f94f84d271246aa9aaef39ef0c7eaac from peer 096fb4c61297434290d3c98ade96da6f at {username='slave'} at 127.22.208.129:41297: session id = 096fb4c61297434290d3c98ade96da6f-7f94f84d271246aa9aaef39ef0c7eaac
I20260812 08:02:50.407332  2451 tablet_copy_source_session.cc:215] T 7f94f84d271246aa9aaef39ef0c7eaac P 1f20f95d7b6d4556ab254e8ed6506085: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 08:02:50.411916  2876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f94f84d271246aa9aaef39ef0c7eaac. 1 dirs total, 0 dirs full, 0 dirs failed
W20260812 08:02:50.409662  2811 tablet_service.cc:409] Invalid argument: UpdateConsensus: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.22.208.131:45579: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" caller_uuid: "1d233039be434b58822feb9300ddbbd8" caller_term: 3 preceding_id { term: 1 index: 34 } ops { id { term: 3 index: 35 } timestamp: 7317593164582821888 op_type: NO_OP noop_request { } } ops { id { term: 3 index: 36 } timestamp: 7317593164907425792 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\257}\262xsI\254\240\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0006\271il\252A\344\304\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\371\317\375i\216R&A \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\201\320\006u\311:\342\2410\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\337\013KkJ\247\335\354@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\336\3271m\226\200\305\370P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000}=\275o3\340EV`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000p\360yf\214z_\244p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\344(Cq0\034\373\324\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000O\t!{w\336q.\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""32a1b5c3ad48a0018f77154b262dedb7b786dc08edaf9d0c2fe2ef33378c2ba91d5f95245c70fb19af2b6d8c184a350bd9475f4b398283c8cabb72bf30d4be34c00b03d5a6516b98b997adcc2a26933c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152271101952 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 178 first_incomplete_seq_no: 168 attempt_no: 73 } } ops { id { term: 3 index: 37 } timestamp: 7317593164956864512 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\002\226mie\322\260\247\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000%\306\273i9Z\200\025\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\" \221~\360\354\370\024 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\343\240S}V\031l,0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\222\314@wDZ\025$@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000<#Qu\220A\364\026P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000y8\245o\233\026+]`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\223\205\325}\002Y\265\277p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d163bda4566ec5cea85ff0b6e568e4859bb3d2b1759ae158f19e9927c6b2136f841238ce909efe982218a5d8806074714d415bd7fbe94fc18ff84544e91738c0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593158782095360 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 180 first_incomplete_seq_no: 168 attempt_no: 50 } } ops { id { term: 3 index: 38 } timestamp: 7317593164974698496 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000D]\"j\377\"I\370\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227?\306\177b\265\022]\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\023\246}h\324<\212\347 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000!\225\177t\247\313\315\3340\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\220\362\'o\333\001\217\204@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227\366\206~\341\374\312SP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000o\243\216\177#S\340\205`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\323\373Yh\233\305\337_p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\266>\265{(\371>\342\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242\273\025m\207\241\275\245\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c7e2170c98ae736553ef8908a61b2e8a78e0eefceed440c75bd7cd3c69c658d7375bef4b697682a32f3f53b0cd1f01be8586090bc577a9abb04e49f147b10bfb0b0bf629fc9c4879fcf742591a4a00a8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164907425792 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 189 first_incomplete_seq_no: 168 attempt_no: 0 } } ops { id { term: 3 index: 39 } timestamp: 7317593165019877376 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\2617j\207i\336G\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000(\322<~\275\311\351\334\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000D\3350x\266\371\310\376 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\310\353\361s\233\222\232\0360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0009i\250li\341R\006@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000`\2308r\243p\023\311P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000e\304\376j\326\260\326\374`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000cw\353mh9\357\336p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\240\355\221f\210\334t\320\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\377\221\370w!\024\007\315\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227}Vt\006\367g\031\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5e4c021c203d179cf8013be53206a839094bd0f821089d190b29d661cf7341a89cd977c01289da18ca5859d75ce46f1e36be5dfe1819d58c0dd1b2e0046d2e6dce1fd803193d686745f89d718a5113352d9936854b94da4c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152160854016 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 168 first_incomplete_seq_no: 168 attempt_no: 74 } } ops { id { term: 3 index: 40 } timestamp: 7317593165025030144 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000Tf\211s\311\350\362>\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000/G\346lI\265\2060\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\025\334\307i!\370z4 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\343M\030l\330\264\200\0360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0001\343\345u)\n\\\235@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\363\335\207\177bm\031!P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000ipary:\004\330`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\350aoii\036\330cp\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242/\217h\377p\303\336\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\010\035pW\231\016\205\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000z\307_y\316\212Of\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\3278\275j\003\205\307\\\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\230\313\020u\207!f\303\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\321\331*\177.\231 \"\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7ddc33b5878de1422aa5a0f8bfd660f6233c22eb023108bfc85b2996b20eb7ed480abfa27356519bbfff60572e1c212866fa2668e1df55527f2044f93f20af9f779c17495179ac4958702d675969a37365fec2b0713f8899bc568f5f2dbd89d78b3adc088a876f9cfa2ae8d52e382b91" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593152167272448 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 171 first_incomplete_seq_no: 168 attempt_no: 74 } } ops { id { term: 3 index: 41 } timestamp: 7317593165036326912 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\306\240\324v\263\254\275\337\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\222\357!|J\234\231\200\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000?\006\312{\332\226\214\207 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\340\220\265|\367\215g50\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\372Rn{\002!\341\276@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000b\177\027{%\023\030`P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000*\215\307v\201&\2238`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9a750aa6741c4bb7d1fa8d25b091445ed860ce4df7109d1e3013777f1c149703f9adf42c96f3663ae4491f5cf95408e77043c7296390db00" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593164974698496 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 192 first_incomplete_seq_no: 168 attempt_no: 1 } } ops { id { term: 3 index: 42 } timestamp: 7317593165105995776 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\220\"\321}\371$s;\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\356\340\315\177eL\377\023\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000no>w\3063\251\374 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\235%2m\237\001Vx0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\242\346Ew /+\232@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0003\340\023yL\365l\274P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\277(3o\214^\265!`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\270\217\363q\334%\247\'p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000.\316\207z\214\222!i\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c6df0e753c5c4db22cda9755060ac8e9b47732147af1b3e91009da0bed946274e9c2e7a1f5ea628b2b359f9f9b056d6c8fb8c48a56de902c0f06df4698592a65b1556a53bd6f528c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165025030144 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 200 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 3 index: 43 } timestamp: 7317593165132206080 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000{m\207v#\010\266n\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0004.\322t\372^\245\234\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\n\332h\177\366\001\313\357 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\325\347\376f\034\307\033\3320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000M\246\244gZ\351\247\t@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000$\000\254f,\2434\177P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000s1OwLF\314|`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034\326\370lm[(\261p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\355C.}\235\034\033\333\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c183d1a46a70cd85a0bfb1fa127b49c4eb918ef9c28369366ce6f0e85ba44cbf433964e5da1e31bb744857c672f545f5e6828287048f01a317a50a9328b28a0b81e7c6762f2cdbe8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165036326912 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 208 first_incomplete_seq_no: 190 attempt_no: 0 } } ops { id { term: 3 index: 44 } timestamp: 7317593165167423488 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\200\333\357m\201\265\\\023\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000u\274\340~v\002\325~\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\275\0178k\376\027p\251 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\251H\205f\2776\022\2220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\324#\345q\230\006\010e@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\016g\006z\035\255\306\365P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\306\205&i\361\230\353x`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000c>\242jg\017\277\226p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000u`\021{\254ag\233\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\352Jl{\342)07\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0006\2627p\347\307\265*\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\355\324\270p\324\204|\317\260\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000,\003dii!\240\006\300\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\027\320yp\034\225\327\355\320\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\306\023\262}%$\324\202\340\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\355X\235|{\2219\376\360\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""136a94177440f7eca08b034d8d0bcb7fe8e233063507c7c4794903fa51485eef3f335eed8b3ab5b2f4f7f74c8719b5252e428917246cb9767deee49e1dc6c1aca0eb8f0c11f8db6dbdf1ce6c9c4eaa1aac16e6a92702efb8e38e02ec00c3cc61d0e9d56f4f76e62b99cb4080ddef831ca099d4a9a6b1a9968402b4e01b61c05d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165105995776 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 210 first_incomplete_seq_no: 195 attempt_no: 0 } } ops { id { term: 3 index: 45 } timestamp: 7317593165170802688 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\235\016yn\344Z\014\003\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000;\005-~\221\030|\033\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227\272\371p\003+u\350 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\352\310\261vK\t\034\2500\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\337L\021hc:\316\243@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\307E\225k*~\326\353P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\333\251\353y\212\010r\350`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\347\377ir[\374\352#p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\374\216Uy\005x\216\362\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000P\206\314r\347D\362\276\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""b31bef07ddd3df36cd1bbeda311dc1deca25d298f24a3e5f4b3c300e38edd9ca38eb8ce2e0dd0ee66166cac94716c681fd312edccff75b749b17d64f73edd4dc6947d453558954a2db886b0db258a2db" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165105995776 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 212 first_incomplete_seq_no: 195 attempt_no: 0 } } ops { id { term: 3 index: 46 } timestamp: 7317593165256589312 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\344-\"n<u\247\227\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\276Jmg\217,A\t\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\362\236\016j\370\312\005\264 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000!\377\243|\210\303+d0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000M\253eii\321\300Q@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\226\226C~,\205\276\377P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\226>\316w\0043}\304`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0003\350\twj\2369\251p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000 9\215yX\036\241=\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000e\334p\177\'\027\227\324\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000Q1wl\2436\\\365\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bc3cbd7fef7a3923159a7db52366024ffdcc073d3bdeaafa5c2e81626260aec4a73eee04a8f8fb21080e7a35dcf2a510a2e31affd8031c2277952c73c532deb5359c65315d6f1eec199b0c15246c87aa81cdeb2dec15f236" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 215 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 3 index: 47 } timestamp: 7317593165270822912 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\264\020\275i\337YMG\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\227\220\312u\033\367\313\306\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\316\2333v\231\264\374\263 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000%\312\230w\026K\216M0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\353\210\230t\016Yf\006@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\312\002nv\370Z\242nP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000-\3573n\227\017(\373`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\342X\367mS\316\374\355p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8b18eae921ccd960af8d51c84de873b7e917b8dea8d45972a859e75c4cc8bb6cc48941af9e6ed6a004a8a49cb79e41f603807ceed1141fc2e07fa56cb88b0de5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 216 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 3 index: 48 } timestamp: 7317593165273231360 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\037B3o\'9m\017\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\247\317\223{2\240\r\351\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\243\277Bh\036\241\242% \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000#\225\361z\276\330|90\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\261\303%h\337r\031\217@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0007\302\316mK\354\020\220P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000r\t\215t\341\260]\'`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000x;\254w\010\366\262ep\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000%\r\206q,q\260f\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\265o\330s\024\334o/\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5f00758c2e4a9475dffc91297d3c564d3d998150d1f1d73701aa44eba32583f68e621054103c0459096db757843facd555875cc3d3b6642680a401289e90147feb50c59d87bbfaeccb1cf3f3b3a7fa17" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7317593165207244800 authz_token { token_data: "\010\321\323\360\323\006\"3\n\005slave\022*\n 3149907d56be4a1294f29fe93c26fa13\020\001\030\001 \001(\001" signature: "<redacted>""\017t\350eu\333\242\344\247=I\021\001\377S\363<\210\220\223\304#\203\263o\277g\265\221\25014%\257\211\272Vz\206\265[Zy\265$\r`\r\010V\351\203\010?\316\021\343\266\353\227\264\001Q\307" signing_key_seq_num: 0 } } request_id { client_id: "3379cf0ebb0c4197a17964803765cfd9" seq_no: 219 first_incomplete_seq_no: 215 attempt_no: 0 } } ops { id { term: 3 index: 49 } timestamp: 7317593165275332608 op_type: WRITE_OP write_request { tablet_id: "071986b2c7af4287bb4a2e6063c28a69" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\372\226xo\025X\230\003\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0002\333\376{\247\020\261\344\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\271\0101wra\\@ \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000g\321\030o\312\202\205@0\000\000\000\000\000\000\00
I20260812 08:02:50.414920  2211 consensus_queue.cc:579] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: The tablet replica hosted on peer 3d5a93c6df6741aa876e18f7cc21185c has failed
I20260812 08:02:50.418159  2278 consensus_queue.cc:237] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 3.129, Last appended by leader: 34, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } }
I20260812 08:02:50.418337  2876 tablet_copy_client.cc:806] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: tablet copy: Starting download of 0 data blocks...
I20260812 08:02:50.418669  2876 tablet_copy_client.cc:670] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: tablet copy: Starting download of 1 WAL segments...
I20260812 08:02:50.419595  2135 raft_consensus.cc:1275] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 129. Preceding OpId from leader: term: 3 index: 130. (index mismatch)
I20260812 08:02:50.420262  2696 consensus_queue.cc:1048] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260812 08:02:50.422004  2876 tablet_copy_client.cc:538] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 08:02:50.423032  2695 raft_consensus.cc:2955] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 3 LEADER]: Committing config change with OpId 3.130: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
I20260812 08:02:50.423552  2135 raft_consensus.cc:2955] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Committing config change with OpId 3.130: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
I20260812 08:02:50.424816  2876 tablet_bootstrap.cc:492] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: Bootstrap starting.
I20260812 08:02:50.425091  1814 catalog_manager.cc:5232] ChangeConfig:REMOVE_PEER RPC for tablet 071986b2c7af4287bb4a2e6063c28a69 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 08:02:50.427337  1818 catalog_manager.cc:5719] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 reported cstate change: config changed from index -1 to 130, VOTER 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129) evicted. New cstate: current_term: 3 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } }
W20260812 08:02:50.430266  2786 tablet_service.cc:409] Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c: from {username='slave'} at 127.0.0.1:39020: tablet_id: "071986b2c7af4287bb4a2e6063c28a69" reason: "TS 3d5a93c6df6741aa876e18f7cc21185c not found in new config with opid_index 130" delete_type: TABLET_DATA_TOMBSTONED dest_uuid: "3d5a93c6df6741aa876e18f7cc21185c" cas_config_opid_index_less_or_equal: -1
W20260812 08:02:50.430716  1813 catalog_manager.cc:5036] TS 3d5a93c6df6741aa876e18f7cc21185c (127.22.208.129:33759): delete failed for tablet 071986b2c7af4287bb4a2e6063c28a69 because the server uuid is wrong. No further retry: Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: 096fb4c61297434290d3c98ade96da6f. Requested UUID: 3d5a93c6df6741aa876e18f7cc21185c
I20260812 08:02:50.431655  2278 consensus_queue.cc:237] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 3.130, Last appended by leader: 34, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } }
I20260812 08:02:50.433055  2135 raft_consensus.cc:1275] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Refusing update from remote peer 1d233039be434b58822feb9300ddbbd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 130. Preceding OpId from leader: term: 3 index: 131. (index mismatch)
W20260812 08:02:50.433221  2211 consensus_peers.cc:597] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 -> Peer 096fb4c61297434290d3c98ade96da6f (127.22.208.129:33759): Couldn't send request to peer 096fb4c61297434290d3c98ade96da6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 071986b2c7af4287bb4a2e6063c28a69. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:02:50.433530  2696 consensus_queue.cc:1048] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20260812 08:02:50.435654  2695 raft_consensus.cc:2955] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 [term 3 LEADER]: Committing config change with OpId 3.131: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.436157  2135 raft_consensus.cc:2955] T 071986b2c7af4287bb4a2e6063c28a69 P 714a507a0a7149e6a58104455bec51f2 [term 3 FOLLOWER]: Committing config change with OpId 3.131: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } } }
I20260812 08:02:50.437745  1814 catalog_manager.cc:5232] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 071986b2c7af4287bb4a2e6063c28a69 with cas_config_opid_index 130: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260812 08:02:50.438426  1827 catalog_manager.cc:5719] T 071986b2c7af4287bb4a2e6063c28a69 P 1d233039be434b58822feb9300ddbbd8 reported cstate change: config changed from index 130 to 131, NON_VOTER 096fb4c61297434290d3c98ade96da6f (127.22.208.129) added. New cstate: current_term: 3 leader_uuid: "1d233039be434b58822feb9300ddbbd8" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "714a507a0a7149e6a58104455bec51f2" member_type: VOTER last_known_addr { host: "127.22.208.130" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d233039be434b58822feb9300ddbbd8" member_type: VOTER last_known_addr { host: "127.22.208.131" port: 39265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "096fb4c61297434290d3c98ade96da6f" member_type: NON_VOTER last_known_addr { host: "127.22.208.129" port: 33759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260812 08:02:50.445596  2876 log.cc:826] T 7f94f84d271246aa9aaef39ef0c7eaac P 096fb4c61297434290d3c98ade96da6f: Log is configured to *not* fsync() on all Append() calls
W20260812 08:02:50.462785 23362 subprocess.cc:278] Child process 2720 (/tmp/dist-test-taskTcd653/build/debug/bin/kudu --fs_wal_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/wal-new --fs_data_dirs=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new --block_manager=log --webserver_interface=localhost --never_fsync --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskTcd653/test-tmp/raft_consensus-itest.1.RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID_1.1786521625833775-23362-0/minicluster-data/ts-0/data-new/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_server_key_size=768 --openssl_security_level_override=0 tserver run --rpc_bind_addresses=127.22.208.129:33759 --local_ip_for_outbound_sockets=127.22.208.129 --webserver_interface=127.22.208.129 --webserver_port=0 --tserver_master_addrs=127.22.208.190:38821 --builtin_ntp_servers=127.22.208.148:44857 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=10 --raft_prepare_replacement_before_eviction=true) was orphaned. Sending signal 9...
I20260812 08:02:50.474824 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 2046
I20260812 08:02:50.484144 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 2192
I20260812 08:02:50.494889 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 2340
I20260812 08:02:50.504894 23362 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskTcd653/build/debug/bin/kudu with pid 1796
2026-08-12T08:02:50Z chronyd exiting
[       OK ] RaftConsensusParamReplicationModesITest.TestRestartWithDifferentUUID/1 (7030 ms)
[----------] 1 test from RaftConsensusParamReplicationModesITest (7030 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 3 test suites ran. (144691 ms total)
[  PASSED  ] 8 tests.
I20260812 08:02:50.537181 23362 logging.cc:436] LogThrottler /workspace/apache/dev/local/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 22 messages since previous log ~2 seconds ago
