Note: This is test shard 3 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from MaintenanceModeRF3ITest
[ RUN      ] MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable
2026-08-12T08:03:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:03:55Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 08:03:55.185552 12649 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskedrhlK/build/debug/bin/kudu
/tmp/dist-test-taskedrhlK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.90.126:45577
--webserver_interface=127.12.90.126
--webserver_port=0
--builtin_ntp_servers=127.12.90.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.90.126:45577 with env {}
W20260812 08:03:55.300194 12661 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:55.300424 12661 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:55.300462 12661 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:55.304023 12661 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:03:55.304088 12661 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:55.304111 12661 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:03:55.304128 12661 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:03:55.308017 12661 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.90.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.90.126:45577
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.90.126:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.90.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:55.309139 12661 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:55.310396 12661 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:55.316200 12671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.316206 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.316210 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.316434 12661 server_base.cc:1061] running on GCE node
I20260812 08:03:55.316910 12661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:55.317870 12661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:55.319058 12661 hybrid_clock.cc:648] HybridClock initialized: now 1786521835319044 us; error 33 us; skew 500 ppm
I20260812 08:03:55.320811 12661 webserver.cc:533] Webserver started at http://127.12.90.126:33555/ using document root <none> and password file <none>
I20260812 08:03:55.321538 12661 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:55.321622 12661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:55.321847 12661 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:55.323366 12661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data/instance:
uuid: "e1308f5d8c854dfbb7171ea656ea1a20"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.323875 12661 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal/instance:
uuid: "e1308f5d8c854dfbb7171ea656ea1a20"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.327255 12661 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20260812 08:03:55.329463 12680 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.330371 12661 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:03:55.330507 12661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data,/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal
uuid: "e1308f5d8c854dfbb7171ea656ea1a20"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.330614 12661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:55.346374 12661 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:55.347066 12661 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:55.347273 12661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:55.355499 12765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.90.126:45577 every 8 connection(s)
I20260812 08:03:55.355509 12661 rpc_server.cc:307] RPC server started. Bound to: 127.12.90.126:45577
I20260812 08:03:55.356634 12661 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/data/info.pb
I20260812 08:03:55.359669 12767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:55.362429 12649 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskedrhlK/build/debug/bin/kudu as pid 12661
I20260812 08:03:55.362622 12649 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/master-0/wal/instance
I20260812 08:03:55.365656 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20: Bootstrap starting.
I20260812 08:03:55.368213 12767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:55.368916 12767 log.cc:826] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20: Log is configured to *not* fsync() on all Append() calls
I20260812 08:03:55.370991 12767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20: No bootstrap required, opened a new log
I20260812 08:03:55.374158 12767 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } }
I20260812 08:03:55.374432 12767 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:55.374589 12767 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1308f5d8c854dfbb7171ea656ea1a20, State: Initialized, Role: FOLLOWER
I20260812 08:03:55.375025 12767 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } }
I20260812 08:03:55.375145 12767 raft_consensus.cc:399] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:55.375185 12767 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:55.375265 12767 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:55.376029 12767 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } }
I20260812 08:03:55.376327 12767 leader_election.cc:304] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1308f5d8c854dfbb7171ea656ea1a20; no voters: 
I20260812 08:03:55.376582 12767 leader_election.cc:290] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:55.376778 12775 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:55.377054 12775 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [term 1 LEADER]: Becoming Leader. State: Replica: e1308f5d8c854dfbb7171ea656ea1a20, State: Running, Role: LEADER
I20260812 08:03:55.377415 12775 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } }
I20260812 08:03:55.377966 12767 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:03:55.379418 12777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } } }
I20260812 08:03:55.379554 12777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:55.380836 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1308f5d8c854dfbb7171ea656ea1a20. Latest consensus state: current_term: 1 leader_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1308f5d8c854dfbb7171ea656ea1a20" member_type: VOTER last_known_addr { host: "127.12.90.126" port: 45577 } } }
I20260812 08:03:55.381002 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:55.381067 12790 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:55.384714 12790 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:55.390364 12790 catalog_manager.cc:1383] Generated new cluster ID: f846758f108c498da04f145265b8a592
I20260812 08:03:55.390452 12790 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:55.410775 12790 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:03:55.411963 12790 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:55.424451 12790 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e1308f5d8c854dfbb7171ea656ea1a20: Generated new TSK 0
I20260812 08:03:55.425177 12790 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:03:55.434342 12649 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskedrhlK/build/debug/bin/kudu
/tmp/dist-test-taskedrhlK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.90.65:0
--local_ip_for_outbound_sockets=127.12.90.65
--webserver_interface=127.12.90.65
--webserver_port=0
--tserver_master_addrs=127.12.90.126:45577
--builtin_ntp_servers=127.12.90.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
W20260812 08:03:55.544970 12804 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:55.545235 12804 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:55.545303 12804 flags.cc:432] Enabled unsafe flag: --enable_log_gc=false
W20260812 08:03:55.545395 12804 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:55.548712 12804 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:55.548899 12804 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.90.65
I20260812 08:03:55.552680 12804 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.90.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.90.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.90.65
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.12.90.126:45577
--never_fsync=true
--heap_profile_path=/tmp/kudu.12804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.90.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:55.553905 12804 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:55.555092 12804 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:03:55.558023 12804 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:03:55.562059 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.562224 12804 server_base.cc:1061] running on GCE node
W20260812 08:03:55.562112 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.562080 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.562772 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:55.563344 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:55.564498 12804 hybrid_clock.cc:648] HybridClock initialized: now 1786521835564483 us; error 32 us; skew 500 ppm
I20260812 08:03:55.566382 12804 webserver.cc:533] Webserver started at http://127.12.90.65:44647/ using document root <none> and password file <none>
I20260812 08:03:55.566942 12804 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:55.566996 12804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:55.567157 12804 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:55.568688 12804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/instance:
uuid: "4ea9c07229134ecda7cfb276629c1f77"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.569130 12804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal/instance:
uuid: "4ea9c07229134ecda7cfb276629c1f77"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.572598 12804 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:03:55.574931 12830 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.576053 12804 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:03:55.576153 12804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data,/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal
uuid: "4ea9c07229134ecda7cfb276629c1f77"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.576234 12804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:55.590974 12804 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:55.591625 12804 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:55.591776 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:55.592343 12804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:55.593425 12804 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:55.593482 12804 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.593523 12804 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:55.593551 12804 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.603121 12804 rpc_server.cc:307] RPC server started. Bound to: 127.12.90.65:32817
I20260812 08:03:55.603240 13013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.90.65:32817 every 8 connection(s)
I20260812 08:03:55.604125 12804 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/info.pb
I20260812 08:03:55.611495 12649 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskedrhlK/build/debug/bin/kudu as pid 12804
I20260812 08:03:55.611620 12649 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal/instance
I20260812 08:03:55.612804 13014 heartbeater.cc:344] Connected to a master server at 127.12.90.126:45577
I20260812 08:03:55.613101 13014 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:55.613739 13014 heartbeater.cc:507] Master 127.12.90.126:45577 requested a full tablet report, sending...
I20260812 08:03:55.614023 12649 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskedrhlK/build/debug/bin/kudu
/tmp/dist-test-taskedrhlK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.90.66:0
--local_ip_for_outbound_sockets=127.12.90.66
--webserver_interface=127.12.90.66
--webserver_port=0
--tserver_master_addrs=127.12.90.126:45577
--builtin_ntp_servers=127.12.90.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20260812 08:03:55.615338 12705 ts_manager.cc:194] Registered new tserver with Master: 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65:32817)
I20260812 08:03:55.616545 12705 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.90.65:47791
W20260812 08:03:55.726064 13017 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:55.726291 13017 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:55.726365 13017 flags.cc:432] Enabled unsafe flag: --enable_log_gc=false
W20260812 08:03:55.726440 13017 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:55.729734 13017 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:55.729910 13017 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.90.66
I20260812 08:03:55.733697 13017 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.90.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.90.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.90.66
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.12.90.126:45577
--never_fsync=true
--heap_profile_path=/tmp/kudu.13017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.90.66

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:55.734989 13017 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:55.736150 13017 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:03:55.739066 13017 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:03:55.742959 13022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.743140 13028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.743000 13023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.743356 13017 server_base.cc:1061] running on GCE node
I20260812 08:03:55.743726 13017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:55.744248 13017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:55.745431 13017 hybrid_clock.cc:648] HybridClock initialized: now 1786521835745408 us; error 35 us; skew 500 ppm
I20260812 08:03:55.747175 13017 webserver.cc:533] Webserver started at http://127.12.90.66:35807/ using document root <none> and password file <none>
I20260812 08:03:55.747720 13017 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:55.747771 13017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:55.747931 13017 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:55.749555 13017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data/instance:
uuid: "22d69231364e40d8ae0f14528ff4f302"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.750022 13017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal/instance:
uuid: "22d69231364e40d8ae0f14528ff4f302"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.753239 13017 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:03:55.755558 13037 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.756675 13017 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:03:55.756826 13017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data,/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal
uuid: "22d69231364e40d8ae0f14528ff4f302"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.756954 13017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:55.768684 13017 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:55.769474 13017 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:55.769685 13017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:55.770298 13017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:55.771471 13017 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:55.771522 13017 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.771595 13017 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:55.771638 13017 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.782533 13017 rpc_server.cc:307] RPC server started. Bound to: 127.12.90.66:38583
I20260812 08:03:55.782651 13228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.90.66:38583 every 8 connection(s)
I20260812 08:03:55.783799 13017 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/data/info.pb
I20260812 08:03:55.791363 12649 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskedrhlK/build/debug/bin/kudu as pid 13017
I20260812 08:03:55.791563 12649 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-1/wal/instance
I20260812 08:03:55.792901 13230 heartbeater.cc:344] Connected to a master server at 127.12.90.126:45577
I20260812 08:03:55.793191 13230 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:55.793826 12649 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskedrhlK/build/debug/bin/kudu
/tmp/dist-test-taskedrhlK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.90.67:0
--local_ip_for_outbound_sockets=127.12.90.67
--webserver_interface=127.12.90.67
--webserver_port=0
--tserver_master_addrs=127.12.90.126:45577
--builtin_ntp_servers=127.12.90.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20260812 08:03:55.793840 13230 heartbeater.cc:507] Master 127.12.90.126:45577 requested a full tablet report, sending...
I20260812 08:03:55.795039 12705 ts_manager.cc:194] Registered new tserver with Master: 22d69231364e40d8ae0f14528ff4f302 (127.12.90.66:38583)
I20260812 08:03:55.795814 12705 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.90.66:59071
W20260812 08:03:55.903549 13236 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:55.903793 13236 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:55.903864 13236 flags.cc:432] Enabled unsafe flag: --enable_log_gc=false
W20260812 08:03:55.903929 13236 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:55.907358 13236 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:55.907527 13236 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.90.67
I20260812 08:03:55.911226 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.90.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.90.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.90.67
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.12.90.126:45577
--never_fsync=true
--heap_profile_path=/tmp/kudu.13236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.90.67

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:55.912370 13236 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:55.913592 13236 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:03:55.916396 13236 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:03:55.920295 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:55.920295 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.920359 13236 server_base.cc:1061] running on GCE node
W20260812 08:03:55.920291 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:55.920848 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:55.921465 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:55.922644 13236 hybrid_clock.cc:648] HybridClock initialized: now 1786521835922619 us; error 45 us; skew 500 ppm
I20260812 08:03:55.924441 13236 webserver.cc:533] Webserver started at http://127.12.90.67:39205/ using document root <none> and password file <none>
I20260812 08:03:55.925040 13236 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:55.925097 13236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:55.925308 13236 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:55.926892 13236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data/instance:
uuid: "5e28626f24bf4133a82c8619d8f25546"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.927313 13236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal/instance:
uuid: "5e28626f24bf4133a82c8619d8f25546"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.930567 13236 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:03:55.932585 13257 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.933633 13236 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:03:55.933750 13236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data,/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal
uuid: "5e28626f24bf4133a82c8619d8f25546"
format_stamp: "Formatted at 2026-08-12 08:03:55 on dist-test-slave-bw88"
I20260812 08:03:55.933842 13236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:55.958417 13236 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:55.959197 13236 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:55.959412 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:55.960093 13236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:55.961203 13236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:55.961279 13236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.961378 13236 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:55.961414 13236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:55.970928 13236 rpc_server.cc:307] RPC server started. Bound to: 127.12.90.67:41861
I20260812 08:03:55.970963 13418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.90.67:41861 every 8 connection(s)
I20260812 08:03:55.971954 13236 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/data/info.pb
I20260812 08:03:55.980870 13419 heartbeater.cc:344] Connected to a master server at 127.12.90.126:45577
I20260812 08:03:55.980918 12649 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskedrhlK/build/debug/bin/kudu as pid 13236
I20260812 08:03:55.981021 12649 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-2/wal/instance
I20260812 08:03:55.981204 13419 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:55.981891 13419 heartbeater.cc:507] Master 127.12.90.126:45577 requested a full tablet report, sending...
I20260812 08:03:55.983161 12704 ts_manager.cc:194] Registered new tserver with Master: 5e28626f24bf4133a82c8619d8f25546 (127.12.90.67:41861)
I20260812 08:03:55.983784 12704 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.90.67:36003
I20260812 08:03:55.995703 12649 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260812 08:03:56.019416 12704 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37486:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260812 08:03:56.020639 12704 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 08:03:56.037819 12912 tablet_service.cc:1511] Processing CreateTablet for tablet d64a7758356d4101be2e015592bcd75d (DEFAULT_TABLE table=test-workload [id=f3eaa429c5ce4366a48d5554b454d020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:56.037819 13125 tablet_service.cc:1511] Processing CreateTablet for tablet d64a7758356d4101be2e015592bcd75d (DEFAULT_TABLE table=test-workload [id=f3eaa429c5ce4366a48d5554b454d020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:56.038955 13125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64a7758356d4101be2e015592bcd75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:56.038955 12912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64a7758356d4101be2e015592bcd75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:56.040325 13325 tablet_service.cc:1511] Processing CreateTablet for tablet d64a7758356d4101be2e015592bcd75d (DEFAULT_TABLE table=test-workload [id=f3eaa429c5ce4366a48d5554b454d020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:56.041199 13325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64a7758356d4101be2e015592bcd75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:56.045642 13459 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Bootstrap starting.
I20260812 08:03:56.046413 13461 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: Bootstrap starting.
I20260812 08:03:56.046998 13460 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Bootstrap starting.
I20260812 08:03:56.047982 13459 tablet_bootstrap.cc:654] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:56.048595 13461 tablet_bootstrap.cc:654] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:56.048853 13460 tablet_bootstrap.cc:654] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:56.048877 13459 log.cc:826] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Log is configured to *not* fsync() on all Append() calls
I20260812 08:03:56.049432 13461 log.cc:826] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: Log is configured to *not* fsync() on all Append() calls
I20260812 08:03:56.049701 13460 log.cc:826] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Log is configured to *not* fsync() on all Append() calls
I20260812 08:03:56.050773 13459 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: No bootstrap required, opened a new log
I20260812 08:03:56.050966 13459 ts_tablet_manager.cc:1403] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.005s
I20260812 08:03:56.050987 13461 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: No bootstrap required, opened a new log
I20260812 08:03:56.051156 13461 ts_tablet_manager.cc:1403] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:03:56.051968 13460 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: No bootstrap required, opened a new log
I20260812 08:03:56.052175 13460 ts_tablet_manager.cc:1403] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20260812 08:03:56.054118 13459 raft_consensus.cc:359] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.054118 13461 raft_consensus.cc:359] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.054327 13459 raft_consensus.cc:385] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:56.054327 13461 raft_consensus.cc:385] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:56.054399 13461 raft_consensus.cc:740] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e28626f24bf4133a82c8619d8f25546, State: Initialized, Role: FOLLOWER
I20260812 08:03:56.054476 13459 raft_consensus.cc:740] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9c07229134ecda7cfb276629c1f77, State: Initialized, Role: FOLLOWER
I20260812 08:03:56.054813 13460 raft_consensus.cc:359] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.054867 13461 consensus_queue.cc:260] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.054867 13459 consensus_queue.cc:260] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.055004 13460 raft_consensus.cc:385] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:56.055083 13460 raft_consensus.cc:740] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d69231364e40d8ae0f14528ff4f302, State: Initialized, Role: FOLLOWER
I20260812 08:03:56.055552 13460 consensus_queue.cc:260] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.055739 13459 ts_tablet_manager.cc:1434] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:03:56.055739 13461 ts_tablet_manager.cc:1434] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:03:56.056145 13419 heartbeater.cc:499] Master 127.12.90.126:45577 was elected leader, sending a full tablet report...
I20260812 08:03:56.056316 13014 heartbeater.cc:499] Master 127.12.90.126:45577 was elected leader, sending a full tablet report...
I20260812 08:03:56.056481 13460 ts_tablet_manager.cc:1434] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Time spent starting tablet: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:03:56.057224 13230 heartbeater.cc:499] Master 127.12.90.126:45577 was elected leader, sending a full tablet report...
I20260812 08:03:56.099090 13470 raft_consensus.cc:493] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:03:56.099306 13470 raft_consensus.cc:515] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.100559 13470 leader_election.cc:290] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e28626f24bf4133a82c8619d8f25546 (127.12.90.67:41861), 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65:32817)
I20260812 08:03:56.105046 12940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d64a7758356d4101be2e015592bcd75d" candidate_uuid: "22d69231364e40d8ae0f14528ff4f302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9c07229134ecda7cfb276629c1f77" is_pre_election: true
I20260812 08:03:56.105046 13356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d64a7758356d4101be2e015592bcd75d" candidate_uuid: "22d69231364e40d8ae0f14528ff4f302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e28626f24bf4133a82c8619d8f25546" is_pre_election: true
W20260812 08:03:56.105243 13015 tablet.cc:2406] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:03:56.105379 13356 raft_consensus.cc:2468] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22d69231364e40d8ae0f14528ff4f302 in term 0.
I20260812 08:03:56.105379 12940 raft_consensus.cc:2468] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22d69231364e40d8ae0f14528ff4f302 in term 0.
W20260812 08:03:56.105512 13015 tablet_replica_mm_ops.cc:318] Log GC is disabled (check --enable_log_gc)
I20260812 08:03:56.105929 13040 leader_election.cc:304] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22d69231364e40d8ae0f14528ff4f302, 4ea9c07229134ecda7cfb276629c1f77; no voters: 
I20260812 08:03:56.106173 13470 raft_consensus.cc:2804] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:03:56.106263 13470 raft_consensus.cc:493] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:03:56.106325 13470 raft_consensus.cc:3060] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:56.107225 13470 raft_consensus.cc:515] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.107592 13470 leader_election.cc:290] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [CANDIDATE]: Term 1 election: Requested vote from peers 5e28626f24bf4133a82c8619d8f25546 (127.12.90.67:41861), 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65:32817)
I20260812 08:03:56.107954 13356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d64a7758356d4101be2e015592bcd75d" candidate_uuid: "22d69231364e40d8ae0f14528ff4f302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e28626f24bf4133a82c8619d8f25546"
I20260812 08:03:56.108079 12940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d64a7758356d4101be2e015592bcd75d" candidate_uuid: "22d69231364e40d8ae0f14528ff4f302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9c07229134ecda7cfb276629c1f77"
I20260812 08:03:56.108100 13356 raft_consensus.cc:3060] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:56.108232 12940 raft_consensus.cc:3060] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:56.109563 13356 raft_consensus.cc:2468] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22d69231364e40d8ae0f14528ff4f302 in term 1.
I20260812 08:03:56.109665 12940 raft_consensus.cc:2468] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22d69231364e40d8ae0f14528ff4f302 in term 1.
I20260812 08:03:56.109910 13040 leader_election.cc:304] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22d69231364e40d8ae0f14528ff4f302, 5e28626f24bf4133a82c8619d8f25546; no voters: 
I20260812 08:03:56.110126 13470 raft_consensus.cc:2804] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:56.110381 13470 raft_consensus.cc:697] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 LEADER]: Becoming Leader. State: Replica: 22d69231364e40d8ae0f14528ff4f302, State: Running, Role: LEADER
I20260812 08:03:56.110699 13470 consensus_queue.cc:237] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } }
I20260812 08:03:56.113987 12704 catalog_manager.cc:5719] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22d69231364e40d8ae0f14528ff4f302 (127.12.90.66). New cstate: current_term: 1 leader_uuid: "22d69231364e40d8ae0f14528ff4f302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 } health_report { overall_health: UNKNOWN } } }
I20260812 08:03:56.127601 12649 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskedrhlK/build/debug/bin/kudu
/tmp/dist-test-taskedrhlK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.90.68:0
--local_ip_for_outbound_sockets=127.12.90.68
--webserver_interface=127.12.90.68
--webserver_port=0
--tserver_master_addrs=127.12.90.126:45577
--builtin_ntp_servers=127.12.90.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20260812 08:03:56.165572 12940 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:03:56.165572 13356 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:03:56.166290 13470 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:03:56.166478 13473 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea9c07229134ecda7cfb276629c1f77" member_type: VOTER last_known_addr { host: "127.12.90.65" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:03:56.173417 13487 mvcc.cc:204] Tried to move back new op lower bound from 7317593440925908992 to 7317593440712204288. Current Snapshot: MvccSnapshot[applied={T|T < 7317593440925908992}]
I20260812 08:03:56.179541 13488 mvcc.cc:204] Tried to move back new op lower bound from 7317593440933408768 to 7317593440712204288. Current Snapshot: MvccSnapshot[applied={T|T < 7317593440925908992}]
I20260812 08:03:56.179792 13490 mvcc.cc:204] Tried to move back new op lower bound from 7317593440933408768 to 7317593440712204288. Current Snapshot: MvccSnapshot[applied={T|T < 7317593440925908992}]
W20260812 08:03:56.223249 13420 tablet_replica_mm_ops.cc:318] Log GC is disabled (check --enable_log_gc)
W20260812 08:03:56.285676 13231 tablet_replica_mm_ops.cc:318] Log GC is disabled (check --enable_log_gc)
W20260812 08:03:56.334128 13485 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:03:56.334456 13485 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:03:56.334519 13485 flags.cc:432] Enabled unsafe flag: --enable_log_gc=false
W20260812 08:03:56.334561 13485 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:03:56.339866 13485 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:03:56.340054 13485 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.90.68
I20260812 08:03:56.345960 13485 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.90.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.90.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.12.90.68
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.12.90.126:45577
--never_fsync=true
--heap_profile_path=/tmp/kudu.13485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.90.68

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:03:56.347508 13485 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:56.349177 13485 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20260812 08:03:56.353152 13485 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than  raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260812 08:03:56.358098 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:56.358481 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:56.359489 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:56.359674 13485 server_base.cc:1061] running on GCE node
I20260812 08:03:56.360090 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:03:56.360769 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:03:56.364043 13485 hybrid_clock.cc:648] HybridClock initialized: now 1786521836363979 us; error 80 us; skew 500 ppm
I20260812 08:03:56.366604 13485 webserver.cc:533] Webserver started at http://127.12.90.68:35635/ using document root <none> and password file <none>
I20260812 08:03:56.367462 13485 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:56.367545 13485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:56.367774 13485 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:56.370033 13485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data/instance:
uuid: "549baf4e4a6b440f85ccb69f2f79899f"
format_stamp: "Formatted at 2026-08-12 08:03:56 on dist-test-slave-bw88"
I20260812 08:03:56.370661 13485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal/instance:
uuid: "549baf4e4a6b440f85ccb69f2f79899f"
format_stamp: "Formatted at 2026-08-12 08:03:56 on dist-test-slave-bw88"
I20260812 08:03:56.380043 13485 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.005s	sys 0.000s
I20260812 08:03:56.389400 13535 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:56.390992 13485 fs_manager.cc:730] Time spent opening block manager: real 0.009s	user 0.001s	sys 0.000s
I20260812 08:03:56.391127 13485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data,/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal
uuid: "549baf4e4a6b440f85ccb69f2f79899f"
format_stamp: "Formatted at 2026-08-12 08:03:56 on dist-test-slave-bw88"
I20260812 08:03:56.391264 13485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:56.438702 13485 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:56.439656 13485 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:56.439888 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:56.440716 13485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:56.442260 13485 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:56.442337 13485 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:56.442402 13485 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:56.442441 13485 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:56.458487 13485 rpc_server.cc:307] RPC server started. Bound to: 127.12.90.68:44551
I20260812 08:03:56.458683 13697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.90.68:44551 every 8 connection(s)
I20260812 08:03:56.459882 13485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/data/info.pb
I20260812 08:03:56.466423 12649 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskedrhlK/build/debug/bin/kudu as pid 13485
I20260812 08:03:56.466544 12649 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-3/wal/instance
I20260812 08:03:56.494526 13698 heartbeater.cc:344] Connected to a master server at 127.12.90.126:45577
I20260812 08:03:56.494858 13698 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:56.495785 13698 heartbeater.cc:507] Master 127.12.90.126:45577 requested a full tablet report, sending...
I20260812 08:03:56.497125 12695 ts_manager.cc:194] Registered new tserver with Master: 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68:44551)
I20260812 08:03:56.497987 12695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.90.68:37525
I20260812 08:03:56.520884 12695 ts_manager.cc:295] Set tserver state for 4ea9c07229134ecda7cfb276629c1f77 to MAINTENANCE_MODE
I20260812 08:03:56.522377 13008 generic_service.cc:175] {username='slave'} at 127.0.0.1:43700 changed flags via RPC: flush_threshold_secs from '120' to '1'
W20260812 08:03:56.523375 13008 generic_service.cc:139] {username='slave'} at 127.0.0.1:43700 forcing change of non-runtime-safe flag env_inject_eio_globs
I20260812 08:03:56.523463 13008 generic_service.cc:175] {username='slave'} at 127.0.0.1:43700 changed flags via RPC: env_inject_eio_globs from '*' to '/tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/**'
W20260812 08:03:56.524232 13008 generic_service.cc:139] {username='slave'} at 127.0.0.1:43700 forcing change of non-runtime-safe flag env_inject_eio
I20260812 08:03:56.524309 13008 generic_service.cc:175] {username='slave'} at 127.0.0.1:43700 changed flags via RPC: env_inject_eio from '0' to '1'
W20260812 08:03:56.528505 13492 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260812 08:03:56.528563 13492 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260812 08:03:56.611354 13009 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260812 08:03:57.106767 13015 maintenance_manager.cc:419] P 4ea9c07229134ecda7cfb276629c1f77: Scheduling FlushMRSOp(d64a7758356d4101be2e015592bcd75d): perf score=0.001900
E20260812 08:03:57.109958 12836 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260812 08:03:57.110132 12836 env_posix.cc:431] I/O error, context: INJECTED FAILURE
W20260812 08:03:57.110363 12836 ts_tablet_manager.cc:1983] error marking data directory as failed: IO error: All dirs have failed: 
I20260812 08:03:57.110482 12836 ts_tablet_manager.cc:1986] Data dir 95d36eb1c186420c8d95c20c25f4dc46 has 1 tablets
I20260812 08:03:57.110543 12836 ts_tablet_manager.cc:1993] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: failing tablet
W20260812 08:03:57.110806 12836 ts_tablet_manager.cc:1979] Data directory is already marked failed.
W20260812 08:03:57.110874 12836 ts_tablet_manager.cc:1979] Data directory is already marked failed.
W20260812 08:03:57.111330 12836 tablet_replica_mm_ops.cc:214] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet d64a7758356d4101be2e015592bcd75d: unable to open output file for column key INT32 NOT NULL: Could not create new log block container at /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/data/data: INJECTED FAILURE: Input/output error (error 5)
I20260812 08:03:57.111416 12836 maintenance_manager.cc:643] P 4ea9c07229134ecda7cfb276629c1f77: FlushMRSOp(d64a7758356d4101be2e015592bcd75d) complete. Timing: real 0.004s	user 0.004s	sys 0.000s Metrics: {"compiler_manager_pool.queue_time_us":239,"thread_start_us":185,"threads_started":2}
W20260812 08:03:57.115505 13703 mvcc.cc:118] aborting op with timestamp 7317593444763205632 in state 0; MVCC is closed
W20260812 08:03:57.116142 12940 raft_consensus.cc:1579] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 205' and following 0 ops. Status for this op: Illegal state: Not initializing new op; the tablet is stopped
I20260812 08:03:57.116325 12940 raft_consensus.cc:1595] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Rejecting Update request from peer 22d69231364e40d8ae0f14528ff4f302 for term 1. Could not prepare a single op due to: Illegal state: Not initializing new op; the tablet is stopped
I20260812 08:03:57.116726 13707 tablet_replica.cc:333] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: stopping tablet replica
I20260812 08:03:57.116998 13707 raft_consensus.cc:2243] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 08:03:57.117224 13707 raft_consensus.cc:2272] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 08:03:57.118059 13040 consensus_peers.cc:597] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 -> Peer 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65:32817): Couldn't send request to peer 4ea9c07229134ecda7cfb276629c1f77. Status: Illegal state: Rejecting Update request from peer 22d69231364e40d8ae0f14528ff4f302 for term 1. Could not prepare a single op due to: Illegal state: Not initializing new op; the tablet is stopped. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:03:57.243736 13040 consensus_queue.cc:579] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: The tablet replica hosted on peer 4ea9c07229134ecda7cfb276629c1f77 has failed
I20260812 08:03:57.247982 13154 consensus_queue.cc:237] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } }
I20260812 08:03:57.252154 13355 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20260812 08:03:57.253376 13470 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 235, Time since last communication: 0.000s
I20260812 08:03:57.256997 13473 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index -1 to 237, VOTER 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } }
I20260812 08:03:57.263496 12684 catalog_manager.cc:5232] ChangeConfig:REMOVE_PEER RPC for tablet d64a7758356d4101be2e015592bcd75d with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260812 08:03:57.264470 12695 catalog_manager.cc:5719] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 reported cstate change: config changed from index -1 to 237, VOTER 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65) evicted. New cstate: current_term: 1 leader_uuid: "22d69231364e40d8ae0f14528ff4f302" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } health_report { overall_health: HEALTHY } } }
I20260812 08:03:57.265821 13356 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index -1 to 237, VOTER 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } }
I20260812 08:03:57.275252 12912 tablet_service.cc:1558] Processing DeleteTablet for tablet d64a7758356d4101be2e015592bcd75d with delete_type TABLET_DATA_TOMBSTONED (TS 4ea9c07229134ecda7cfb276629c1f77 not found in new config with opid_index 237) from {username='slave'} at 127.0.0.1:43684
I20260812 08:03:57.276410 13154 consensus_queue.cc:237] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.242, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } }
I20260812 08:03:57.275933 13714 ts_tablet_manager.cc:1916] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260812 08:03:57.279245 13356 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 244. (index mismatch)
I20260812 08:03:57.279829 13515 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 240, Time since last communication: 0.000s
I20260812 08:03:57.282298 13470 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 LEADER]: Committing config change with OpId 1.243: config changed from index 237 to 243, NON_VOTER 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } } }
I20260812 08:03:57.282856 13356 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Committing config change with OpId 1.243: config changed from index 237 to 243, NON_VOTER 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } } }
I20260812 08:03:57.285208 12684 catalog_manager.cc:5232] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d64a7758356d4101be2e015592bcd75d with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260812 08:03:57.289002 12696 catalog_manager.cc:5719] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 reported cstate change: config changed from index 237 to 243, NON_VOTER 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) added. New cstate: current_term: 1 leader_uuid: "22d69231364e40d8ae0f14528ff4f302" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260812 08:03:57.291049 13040 consensus_peers.cc:597] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 -> Peer 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68:44551): Couldn't send request to peer 549baf4e4a6b440f85ccb69f2f79899f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64a7758356d4101be2e015592bcd75d. This is attempt 1: this message will repeat every 5th retry.
I20260812 08:03:57.296072 13714 ts_tablet_manager.cc:1929] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.204
I20260812 08:03:57.296270 13714 log.cc:1199] T d64a7758356d4101be2e015592bcd75d P 4ea9c07229134ecda7cfb276629c1f77: Deleting WAL directory at /tmp/dist-test-taskedrhlK/test-tmp/maintenance_mode-itest.2.MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable.1786521835153608-12649-0/minicluster-data/ts-0/wal/wals/d64a7758356d4101be2e015592bcd75d
I20260812 08:03:57.297598 12681 catalog_manager.cc:5050] TS 4ea9c07229134ecda7cfb276629c1f77 (127.12.90.65:32817): tablet d64a7758356d4101be2e015592bcd75d (table test-workload [id=f3eaa429c5ce4366a48d5554b454d020]) successfully deleted
I20260812 08:03:57.375068 13723 ts_tablet_manager.cc:933] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Initiating tablet copy from peer 22d69231364e40d8ae0f14528ff4f302 (127.12.90.66:38583)
I20260812 08:03:57.375787 13723 tablet_copy_client.cc:323] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: tablet copy: Beginning tablet copy session from remote peer at address 127.12.90.66:38583
I20260812 08:03:57.401533 13187 tablet_copy_service.cc:140] P 22d69231364e40d8ae0f14528ff4f302: Received BeginTabletCopySession request for tablet d64a7758356d4101be2e015592bcd75d from peer 549baf4e4a6b440f85ccb69f2f79899f ({username='slave'} at 127.12.90.68:48565)
I20260812 08:03:57.401721 13187 tablet_copy_service.cc:161] P 22d69231364e40d8ae0f14528ff4f302: Beginning new tablet copy session on tablet d64a7758356d4101be2e015592bcd75d from peer 549baf4e4a6b440f85ccb69f2f79899f at {username='slave'} at 127.12.90.68:48565: session id = 549baf4e4a6b440f85ccb69f2f79899f-d64a7758356d4101be2e015592bcd75d
I20260812 08:03:57.403293 13187 tablet_copy_source_session.cc:215] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 08:03:57.407341 13723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64a7758356d4101be2e015592bcd75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:57.414731 13723 tablet_copy_client.cc:806] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: tablet copy: Starting download of 0 data blocks...
I20260812 08:03:57.414968 13723 tablet_copy_client.cc:670] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: tablet copy: Starting download of 1 WAL segments...
I20260812 08:03:57.426977 13723 tablet_copy_client.cc:538] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 08:03:57.429476 13723 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Bootstrap starting.
I20260812 08:03:57.500608 13698 heartbeater.cc:499] Master 127.12.90.126:45577 was elected leader, sending a full tablet report...
I20260812 08:03:57.507586 13723 log.cc:826] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Log is configured to *not* fsync() on all Append() calls
W20260812 08:03:57.904376 13040 consensus_peers.cc:597] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 -> Peer 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68:44551): Couldn't send request to peer 549baf4e4a6b440f85ccb69f2f79899f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260812 08:03:58.123989 13723 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Bootstrap replayed 1/1 log segments. Stats: ops{read=265 overwritten=0 applied=264 ignored=0} inserts{seen=13050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260812 08:03:58.124536 13723 tablet_bootstrap.cc:492] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Bootstrap complete.
I20260812 08:03:58.125072 13723 ts_tablet_manager.cc:1403] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Time spent bootstrapping tablet: real 0.696s	user 0.600s	sys 0.016s
I20260812 08:03:58.127499 13723 raft_consensus.cc:359] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } }
I20260812 08:03:58.128963 13723 raft_consensus.cc:740] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 549baf4e4a6b440f85ccb69f2f79899f, State: Initialized, Role: LEARNER
I20260812 08:03:58.129447 13723 consensus_queue.cc:260] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 264, Last appended: 1.265, Last appended by leader: 265, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: NON_VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: true } }
I20260812 08:03:58.130450 13723 ts_tablet_manager.cc:1434] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260812 08:03:58.133832 13187 tablet_copy_service.cc:342] P 22d69231364e40d8ae0f14528ff4f302: Request end of tablet copy session 549baf4e4a6b440f85ccb69f2f79899f-d64a7758356d4101be2e015592bcd75d received from {username='slave'} at 127.12.90.68:48565
I20260812 08:03:58.133951 13187 tablet_copy_service.cc:434] P 22d69231364e40d8ae0f14528ff4f302: ending tablet copy session 549baf4e4a6b440f85ccb69f2f79899f-d64a7758356d4101be2e015592bcd75d on tablet d64a7758356d4101be2e015592bcd75d with peer 549baf4e4a6b440f85ccb69f2f79899f
I20260812 08:03:58.213400 13631 raft_consensus.cc:1217] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.242->[1.243-1.463]   Dedup: 1.265->[1.266-1.463]
W20260812 08:03:58.231802 13699 tablet_replica_mm_ops.cc:318] Log GC is disabled (check --enable_log_gc)
I20260812 08:03:58.280344 13473 raft_consensus.cc:1064] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302: attempting to promote NON_VOTER 549baf4e4a6b440f85ccb69f2f79899f to VOTER
I20260812 08:03:58.280766 13473 consensus_queue.cc:237] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 477, Committed index: 477, Last appended: 1.479, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 480 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false } }
I20260812 08:03:58.283798 13355 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 1 index: 479. Preceding OpId from leader: term: 1 index: 480. (index mismatch)
I20260812 08:03:58.284242 13514 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 480, Last known committed idx: 477, Time since last communication: 0.000s
I20260812 08:03:58.290489 13629 raft_consensus.cc:1275] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [term 1 LEARNER]: Refusing update from remote peer 22d69231364e40d8ae0f14528ff4f302: Log matching property violated. Preceding OpId in replica: term: 1 index: 479. Preceding OpId from leader: term: 1 index: 480. (index mismatch)
I20260812 08:03:58.290525 13470 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [term 1 LEADER]: Committing config change with OpId 1.480: config changed from index 243 to 480, 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) changed from NON_VOTER to VOTER. New config: { opid_index: 480 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false } } }
I20260812 08:03:58.294314 13473 consensus_queue.cc:1048] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 480, Last known committed idx: 477, Time since last communication: 0.000s
I20260812 08:03:58.299774 13355 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 5e28626f24bf4133a82c8619d8f25546 [term 1 FOLLOWER]: Committing config change with OpId 1.480: config changed from index 243 to 480, 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) changed from NON_VOTER to VOTER. New config: { opid_index: 480 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false } } }
I20260812 08:03:58.300073 13631 raft_consensus.cc:2955] T d64a7758356d4101be2e015592bcd75d P 549baf4e4a6b440f85ccb69f2f79899f [term 1 FOLLOWER]: Committing config change with OpId 1.480: config changed from index 243 to 480, 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) changed from NON_VOTER to VOTER. New config: { opid_index: 480 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false } } }
I20260812 08:03:58.303968 12695 catalog_manager.cc:5719] T d64a7758356d4101be2e015592bcd75d P 22d69231364e40d8ae0f14528ff4f302 reported cstate change: config changed from index 243 to 480, 549baf4e4a6b440f85ccb69f2f79899f (127.12.90.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "22d69231364e40d8ae0f14528ff4f302" committed_config { opid_index: 480 OBSOLETE_local: false peers { permanent_uuid: "22d69231364e40d8ae0f14528ff4f302" member_type: VOTER last_known_addr { host: "127.12.90.66" port: 38583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e28626f24bf4133a82c8619d8f25546" member_type: VOTER last_known_addr { host: "127.12.90.67" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "549baf4e4a6b440f85ccb69f2f79899f" member_type: VOTER last_known_addr { host: "127.12.90.68" port: 44551 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260812 08:03:58.737572 12649 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskedrhlK/build/debug/bin/kudu with pid 12804
I20260812 08:03:58.752197 12649 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskedrhlK/build/debug/bin/kudu with pid 13017
I20260812 08:03:58.771472 12649 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskedrhlK/build/debug/bin/kudu with pid 13236
I20260812 08:03:58.784678 12649 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskedrhlK/build/debug/bin/kudu with pid 13485
I20260812 08:03:58.799988 12649 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskedrhlK/build/debug/bin/kudu with pid 12661
2026-08-12T08:03:58Z chronyd exiting
[       OK ] MaintenanceModeRF3ITest.TestMaintenanceModeDoesntObstructFailedUnrecoverable (3669 ms)
[----------] 1 test from MaintenanceModeRF3ITest (3669 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (3669 ms total)
[  PASSED  ] 1 test.
