JUnit version 4.13.2
.03:07:03.548 [INFO - main] (RandomUtils.java:49) Using random seed: 1786504023542
03:07:03.612 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
03:07:03.697 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
03:07:03.726 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskF7HWRV/test-tmp
03:07:03.729 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/debug/bin
03:07:03.736 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskF7HWRV/java/build/dist-test/../../../build/debug/bin/kudu, test, mini_cluster, --serialization=pb]
03:07:03.993 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

03:07:04.006 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
03:07:04.007 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

03:07:04.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.008545 10742 mini_oidc.cc:152] Starting JWKS server
03:07:04.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.010871 10742 webserver.cc:460] Webserver started at http://127.0.0.1:35647/ using document root <none> and password file <none>
03:07:04.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.011020 10742 mini_oidc.cc:189] Starting OIDC Discovery server
03:07:04.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.011297 10742 webserver.cc:460] Webserver started at http://127.0.0.1:33789/ using document root <none> and password file <none>
03:07:04.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T03:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
03:07:04.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T03:07:04Z Disabled control of system clock
03:07:04.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.029690 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/logs
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data/info.pb
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:04.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.190:42983
03:07:04.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.190
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json with env {}
03:07:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.153070 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.153333 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.153378 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:04.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157531 10754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:07:04.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157588 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:04.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157606 10754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:07:04.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157624 10754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:07:04.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157639 10754 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:07:04.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.157652 10754 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35647/jwks.json
03:07:04.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.162400 10754 master_runner.cc:386] Master server non-default flags:
03:07:04.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.190:42983
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data/info.pb
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.190
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:04.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10754
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/logs
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:04.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.163715 10754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:04.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.165150 10754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:04.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.174875 10762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.174856 10760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.174860 10759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.174943 10754 server_base.cc:1034] running on GCE node
03:07:04.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.176060 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:04.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.177173 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:04.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.178398 10754 hybrid_clock.cc:648] HybridClock initialized: now 1786504024178340 us; error 78 us; skew 500 ppm
03:07:04.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.185616 10754 webserver.cc:460] Webserver started at http://127.10.125.190:40729/ using document root <none> and password file <none>
03:07:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.186246 10754 fs_manager.cc:359] Metadata directory not provided
03:07:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.186336 10754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.186533 10754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:04.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.188576 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data/instance:
03:07:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "227faf0c829a4e83ac081987be322efd"
03:07:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.189144 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal/instance:
03:07:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "227faf0c829a4e83ac081987be322efd"
03:07:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.192974 10754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
03:07:04.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.195508 10770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:04.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.196872 10754 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:04.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.197088 10754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "227faf0c829a4e83ac081987be322efd"
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.197206 10754 fs_report.cc:383] FS layout report
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal
03:07:04.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data/data
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:04.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.227752 10754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:04.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.228696 10754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:04.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.228940 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:04.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.240442 10754 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.190:42983
03:07:04.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.240485 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.190:42983 every 8 connection(s)
03:07:04.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.241643 10754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/data/info.pb
03:07:04.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.245450 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:04.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.247766 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 10754
03:07:04.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.247933 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-0/wal/instance
03:07:04.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.249478 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/logs
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data/info.pb
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:04.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.189:33083
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.189
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json with env {}
03:07:04.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.249377 10838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.265056 10838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.265991 10773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.189:33083: connect: Connection refused (error 111)
03:07:04.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.267088 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.189:33083: Network error: Client connection negotiation failed: client connection to 127.10.125.189:33083: connect: Connection refused (error 111)
03:07:04.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.316361 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } attempt: 1
03:07:04.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.318005 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.189:33083: Network error: Client connection negotiation failed: client connection to 127.10.125.189:33083: connect: Connection refused (error 111)
03:07:04.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.386245 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } attempt: 2
03:07:04.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.388068 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.189:33083: Network error: Client connection negotiation failed: client connection to 127.10.125.189:33083: connect: Connection refused (error 111)
03:07:04.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.392081 10841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:04.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.392364 10841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:04.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.392409 10841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:04.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397233 10841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:07:04.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397290 10841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:04.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397316 10841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:07:04.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397336 10841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:07:04.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397354 10841 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:07:04.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.397403 10841 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35647/jwks.json
03:07:04.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.403685 10841 master_runner.cc:386] Master server non-default flags:
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.189:33083
03:07:04.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data/info.pb
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.189
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10841
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/logs
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:04.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:04.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:04.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.405005 10841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:04.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.406443 10841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:04.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.416239 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.416335 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.416280 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.416272 10841 server_base.cc:1034] running on GCE node
03:07:04.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.417033 10841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:04.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.418149 10841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:04.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.419322 10841 hybrid_clock.cc:648] HybridClock initialized: now 1786504024419291 us; error 50 us; skew 500 ppm
03:07:04.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.421543 10841 webserver.cc:460] Webserver started at http://127.10.125.189:43329/ using document root <none> and password file <none>
03:07:04.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.422159 10841 fs_manager.cc:359] Metadata directory not provided
03:07:04.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.422226 10841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:04.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.422402 10841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:04.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.424454 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data/instance:
03:07:04.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a7fb453b87e4166ae26f967cb15aab8"
03:07:04.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.425102 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal/instance:
03:07:04.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a7fb453b87e4166ae26f967cb15aab8"
03:07:04.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.429004 10841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
03:07:04.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.432004 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:04.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.433458 10841 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:04.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.433575 10841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a7fb453b87e4166ae26f967cb15aab8"
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.433660 10841 fs_report.cc:383] FS layout report
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data/data
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:04.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:04.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:04.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.449195 10841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:04.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.449995 10841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:04.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.450197 10841 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:04.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.460630 10841 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.189:33083
03:07:04.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.460724 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.189:33083 every 8 connection(s)
03:07:04.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.461611 10841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/data/info.pb
03:07:04.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.464875 10922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:04.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.468726 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 10841
03:07:04.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.468863 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-1/wal/instance
03:07:04.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.468693 10922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.469558 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:04.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/logs
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data/info.pb
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:04.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.188:32999
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.188
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json with env {}
03:07:04.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.484779 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } attempt: 3
03:07:04.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.489061 10922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.494573 10838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.495126 10922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.495925 10863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.496002 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.496960 10922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.527189 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 1
03:07:04.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.528915 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.546237 10922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 1
03:07:04.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.547927 10922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.604161 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 2
03:07:04.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.606103 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.616117 10922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 2
03:07:04.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.617654 10922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.619575 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:04.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.619830 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:04.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.619870 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624131 10924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624181 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624199 10924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624215 10924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624231 10924 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:07:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.624245 10924 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35647/jwks.json
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.629170 10924 master_runner.cc:386] Master server non-default flags:
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.188:32999
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35647/jwks.json
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data/info.pb
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.188
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10924
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/logs
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.630630 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:04.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.632675 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:04.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.643072 10937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.643074 10936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.643505 10924 server_base.cc:1034] running on GCE node
03:07:04.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.643074 10939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:04.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.644340 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:04.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.645887 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:04.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.647333 10924 hybrid_clock.cc:648] HybridClock initialized: now 1786504024647261 us; error 94 us; skew 500 ppm
03:07:04.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.649537 10924 webserver.cc:460] Webserver started at http://127.10.125.188:32817/ using document root <none> and password file <none>
03:07:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.650291 10924 fs_manager.cc:359] Metadata directory not provided
03:07:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.650388 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:04.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.650633 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.652974 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data/instance:
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5a808b17843a4444814cb38e788396f4"
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.653559 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal/instance:
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5a808b17843a4444814cb38e788396f4"
03:07:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.657477 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
03:07:04.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.660079 10949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:04.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.661428 10924 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:04.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.661531 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal
03:07:04.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5a808b17843a4444814cb38e788396f4"
03:07:04.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:04 on dist-test-slave-5k54"
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.661638 10924 fs_report.cc:383] FS layout report
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data/data
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:04.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:04.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.697633 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:04.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.698657 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:04.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.698853 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:04.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.705335 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 3
03:07:04.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.707122 10838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.708879 10922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 3
03:07:04.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.710434 10922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.125.188:32999: Network error: Client connection negotiation failed: client connection to 127.10.125.188:32999: connect: Connection refused (error 111)
03:07:04.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.711292 10924 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.188:32999
03:07:04.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.711467 11003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.188:32999 every 8 connection(s)
03:07:04.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.712414 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/data/info.pb
03:07:04.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.716085 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:04.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.720098 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 10924
03:07:04.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.720196 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/master-2/wal/instance
03:07:04.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.719884 11004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.732060 11004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.737828 11004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } has no permanent_uuid. Determining permanent_uuid...
03:07:04.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.746265 11004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: Bootstrap starting.
03:07:04.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.749338 11004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: Neither blocks nor log segments found. Creating new log.
03:07:04.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.750502 11004 log.cc:824] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: Log is configured to *not* fsync() on all Append() calls
03:07:04.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.752774 11004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: No bootstrap required, opened a new log
03:07:04.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.757108 11004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.757313 11004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:04.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.757359 11004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a808b17843a4444814cb38e788396f4, State: Initialized, Role: FOLLOWER
03:07:04.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.757740 11004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.758174 11012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:04.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.758296 11012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: This master's current role is: FOLLOWER
03:07:04.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.759033 11004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: configured and running, proceeding with master startup.
03:07:04.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.765256 11024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:07:04.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.766855 11024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:07:04.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.763290 10924 master_runner.cc:186] Error getting master registration for 127.10.125.190:42983: OK, instance_id { permanent_uuid: "227faf0c829a4e83ac081987be322efd" instance_seqno: 1786504024237685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
03:07:04.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.767606 10924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:07:04.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.853639 10922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 4
03:07:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.861521 10922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: Bootstrap starting.
03:07:04.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.864220 10922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: Neither blocks nor log segments found. Creating new log.
03:07:04.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.865455 10922 log.cc:824] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: Log is configured to *not* fsync() on all Append() calls
03:07:04.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.867962 10922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: No bootstrap required, opened a new log
03:07:04.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.871343 10838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } attempt: 4
03:07:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.873750 10922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.882637 10838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: Bootstrap starting.
03:07:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.883687 10922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.883817 10922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7fb453b87e4166ae26f967cb15aab8, State: Initialized, Role: FOLLOWER
03:07:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.884456 10922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.886478 10838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: Neither blocks nor log segments found. Creating new log.
03:07:04.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.886606 10922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: configured and running, proceeding with master startup.
03:07:04.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.888375 10838 log.cc:824] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: Log is configured to *not* fsync() on all Append() calls
03:07:04.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.890352 11028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:04.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.890774 11028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: This master's current role is: FOLLOWER
03:07:04.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.892619 10838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: No bootstrap required, opened a new log
03:07:04.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.891018 10841 master_runner.cc:186] Error getting master registration for 127.10.125.190:42983: OK, instance_id { permanent_uuid: "227faf0c829a4e83ac081987be322efd" instance_seqno: 1786504024237685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
03:07:04.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.895668 10841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:07:04.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.898747 10838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.899084 10838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:04.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.899159 10838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227faf0c829a4e83ac081987be322efd, State: Initialized, Role: FOLLOWER
03:07:04.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.899739 10838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:04.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.900414 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:04.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.901284 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: This master's current role is: FOLLOWER
03:07:04.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.901402 10838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: configured and running, proceeding with master startup.
03:07:04.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.911206 11055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:07:04.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.911620 11055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:07:04.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.913074 11041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:07:04.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.913393 11041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:07:04.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.912382 10754 master_runner.cc:263] No leader master found from master 227faf0c829a4e83ac081987be322efd
03:07:04.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:04.913926 10754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:07:04.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.951032 10955 tablet.cc:2369] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:04.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.951799 10866 tablet.cc:2369] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:04.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:04.985409 10777 tablet.cc:2369] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:05.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.028898 11012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:07:05.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.029132 11012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.030413 11012 leader_election.cc:288] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227faf0c829a4e83ac081987be322efd (127.10.125.190:42983), 8a7fb453b87e4166ae26f967cb15aab8 (127.10.125.189:33083)
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.031122 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a808b17843a4444814cb38e788396f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7fb453b87e4166ae26f967cb15aab8" is_pre_election: true
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.031137 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a808b17843a4444814cb38e788396f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227faf0c829a4e83ac081987be322efd" is_pre_election: true
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.031471 10891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a808b17843a4444814cb38e788396f4 in term 0.
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.031544 10807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a808b17843a4444814cb38e788396f4 in term 0.
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.031908 10951 leader_election.cc:302] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a808b17843a4444814cb38e788396f4, 8a7fb453b87e4166ae26f967cb15aab8; no voters: 
03:07:05.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.032423 11012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
03:07:05.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.032502 11012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
03:07:05.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.032545 11012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 0 FOLLOWER]: Advancing to term 1
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.033928 11012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.034268 11012 leader_election.cc:288] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [CANDIDATE]: Term 1 election: Requested vote from peers 227faf0c829a4e83ac081987be322efd (127.10.125.190:42983), 8a7fb453b87e4166ae26f967cb15aab8 (127.10.125.189:33083)
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.034734 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a808b17843a4444814cb38e788396f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a7fb453b87e4166ae26f967cb15aab8"
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.034865 10891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 0 FOLLOWER]: Advancing to term 1
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.035938 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a808b17843a4444814cb38e788396f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227faf0c829a4e83ac081987be322efd"
03:07:05.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.036149 10807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 0 FOLLOWER]: Advancing to term 1
03:07:05.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.036389 10891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a808b17843a4444814cb38e788396f4 in term 1.
03:07:05.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.036761 10951 leader_election.cc:302] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a808b17843a4444814cb38e788396f4, 8a7fb453b87e4166ae26f967cb15aab8; no voters: 
03:07:05.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.038326 11012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 1 FOLLOWER]: Leader election won for term 1
03:07:05.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.038614 11012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a808b17843a4444814cb38e788396f4, State: Running, Role: LEADER
03:07:05.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.040222 11012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } }
03:07:05.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.042677 11012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a808b17843a4444814cb38e788396f4. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.042793 11012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: This master's current role is: LEADER
03:07:05.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.043669 11059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
03:07:05.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.043866 10807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a808b17843a4444814cb38e788396f4 in term 1.
03:07:05.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.045034 11059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
03:07:05.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.052484 10891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [term 1 FOLLOWER]: Refusing update from remote peer 5a808b17843a4444814cb38e788396f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:07:05.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.053179 11012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
03:07:05.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.057953 11028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a808b17843a4444814cb38e788396f4. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.058084 11028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: This master's current role is: FOLLOWER
03:07:05.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.062801 10807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [term 1 FOLLOWER]: Refusing update from remote peer 5a808b17843a4444814cb38e788396f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:07:05.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.063756 11057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
03:07:05.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.064216 11012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.064332 11012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: This master's current role is: LEADER
03:07:05.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.065166 11028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.065291 11028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8 [sys.catalog]: This master's current role is: FOLLOWER
03:07:05.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.065724 11060 mvcc.cc:205] Tried to move back new op lower bound from 7317520486606331904 to 7317520486570041344. Current Snapshot: MvccSnapshot[applied={T|T < 7317520486606331904}]
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.067914 11059 catalog_manager.cc:1347] Generated new cluster ID: 13eaa47b2e5645c29a9de1befe2915b5
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.067971 11059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.072070 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a808b17843a4444814cb38e788396f4. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.072193 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: This master's current role is: FOLLOWER
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.072270 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.072326 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd [sys.catalog]: This master's current role is: FOLLOWER
03:07:05.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.080682 11059 catalog_manager.cc:1370] Generated new certificate authority record
03:07:05.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.082173 11059 catalog_manager.cc:1504] Loading token signing keys...
03:07:05.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.082397 11068 mvcc.cc:205] Tried to move back new op lower bound from 7317520486606331904 to 7317520486570041344. Current Snapshot: MvccSnapshot[applied={T|T < 7317520486606331904}]
03:07:05.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.082798 11061 mvcc.cc:205] Tried to move back new op lower bound from 7317520486606331904 to 7317520486570041344. Current Snapshot: MvccSnapshot[applied={T|T < 7317520486606331904}]
03:07:05.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.087832 11065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a808b17843a4444814cb38e788396f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "227faf0c829a4e83ac081987be322efd" member_type: VOTER last_known_addr { host: "127.10.125.190" port: 42983 } } peers { permanent_uuid: "8a7fb453b87e4166ae26f967cb15aab8" member_type: VOTER last_known_addr { host: "127.10.125.189" port: 33083 } } peers { permanent_uuid: "5a808b17843a4444814cb38e788396f4" member_type: VOTER last_known_addr { host: "127.10.125.188" port: 32999 } } }
03:07:05.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.087986 11065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4 [sys.catalog]: This master's current role is: LEADER
03:07:05.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.097466 11059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a808b17843a4444814cb38e788396f4: Generated new TSK 0
03:07:05.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.098299 11059 catalog_manager.cc:1514] Initializing in-progress tserver states...
03:07:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.110672 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:0
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:05.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:05.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.248041 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:05.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.248312 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:05.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.248364 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:05.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.253023 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:05.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.253158 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.129
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.258057 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:0
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11076
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:05.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.259461 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:05.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.261054 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:05.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.271673 11083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.271718 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.271773 11082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.271930 11076 server_base.cc:1034] running on GCE node
03:07:05.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.272925 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:05.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.273607 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:05.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.274814 11076 hybrid_clock.cc:648] HybridClock initialized: now 1786504025274788 us; error 45 us; skew 500 ppm
03:07:05.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.277021 11076 webserver.cc:460] Webserver started at http://127.10.125.129:44295/ using document root <none> and password file <none>
03:07:05.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.277665 11076 fs_manager.cc:359] Metadata directory not provided
03:07:05.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.277768 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:05.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.278033 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:05.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.280330 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/instance:
03:07:05.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:05.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.281037 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal/instance:
03:07:05.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:05.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.285046 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
03:07:05.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.287926 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.289296 11076 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
03:07:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.289434 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.289568 11076 fs_report.cc:383] FS layout report
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/data
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:05.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:05.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.302421 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:05.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.303244 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:05.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.303490 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.304216 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.305305 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:07:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.305387 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
03:07:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.305444 11076 ts_tablet_manager.cc:610] Registered 0 tablets
03:07:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.305467 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
03:07:05.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.318573 11076 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.129:46115
03:07:05.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.318679 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.129:46115 every 8 connection(s)
03:07:05.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.319590 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:05.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.329532 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 11076
03:07:05.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.329654 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal/instance
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.333630 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:05.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:0
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.335667 11226 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.335963 11226 heartbeater.cc:464] Registering TS with master...
03:07:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.336262 11227 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.336390 11227 heartbeater.cc:464] Registering TS with master...
03:07:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.336645 11226 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:05.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.336644 11227 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:05.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.338469 10966 ts_manager.cc:194] Registered new tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:05.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.339861 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.129:53377
03:07:05.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.341137 10789 ts_manager.cc:194] Registered new tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:05.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.342628 11228 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:05.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.342723 11228 heartbeater.cc:464] Registering TS with master...
03:07:05.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.342929 11228 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:05.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.344403 10878 ts_manager.cc:194] Registered new tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:05.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.473855 11237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:05.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.474100 11237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:05.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.474135 11237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:05.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.478116 11237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:05.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.478204 11237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.130
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.483320 11237 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:0
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11237
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.484408 11237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:05.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.485905 11237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:05.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.497102 11247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.497079 11245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.497119 11244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.497485 11237 server_base.cc:1034] running on GCE node
03:07:05.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.498163 11237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:05.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.498939 11237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:05.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.500114 11237 hybrid_clock.cc:648] HybridClock initialized: now 1786504025500101 us; error 30 us; skew 500 ppm
03:07:05.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.502507 11237 webserver.cc:460] Webserver started at http://127.10.125.130:42263/ using document root <none> and password file <none>
03:07:05.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.503159 11237 fs_manager.cc:359] Metadata directory not provided
03:07:05.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.503232 11237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:05.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.503525 11237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.505755 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/instance:
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.506282 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal/instance:
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.511538 11237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
03:07:05.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.514825 11254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.516355 11237 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.516479 11237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.516572 11237 fs_report.cc:383] FS layout report
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/data
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:05.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:05.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.535531 11237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:05.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.536373 11237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:05.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.536553 11237 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:05.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.537225 11237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:05.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.538477 11237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:07:05.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.538558 11237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
03:07:05.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.538605 11237 ts_tablet_manager.cc:610] Registered 0 tablets
03:07:05.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.538623 11237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
03:07:05.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.552589 11237 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.130:32851
03:07:05.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.552611 11381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.130:32851 every 8 connection(s)
03:07:05.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.553673 11237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:05.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.563714 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 11237
03:07:05.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.563862 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal/instance
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.566744 11383 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.567011 11383 heartbeater.cc:464] Registering TS with master...
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.567660 11384 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.567687 11383 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.567749 11384 heartbeater.cc:464] Registering TS with master...
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.567930 11384 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.568550 11382 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.568634 11382 heartbeater.cc:464] Registering TS with master...
03:07:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.568782 11382 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.568915 10789 ts_manager.cc:194] Registered new tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.569235 10877 ts_manager.cc:194] Registered new tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.569473 10966 ts_manager.cc:194] Registered new tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.569787 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:0
03:07:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.570295 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.130:49423
03:07:05.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.708276 11389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:05.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.708554 11389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:05.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.708590 11389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:05.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.712649 11389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:05.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.712738 11389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.131
03:07:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.717633 11389 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:0
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11389
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:05.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:05.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.718604 11389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:05.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.720104 11389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:05.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.730422 11394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.730479 11399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:05.730428 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.730470 11389 server_base.cc:1034] running on GCE node
03:07:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.731067 11389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.731738 11389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:05.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.732923 11389 hybrid_clock.cc:648] HybridClock initialized: now 1786504025732894 us; error 46 us; skew 500 ppm
03:07:05.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.735154 11389 webserver.cc:460] Webserver started at http://127.10.125.131:38821/ using document root <none> and password file <none>
03:07:05.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.735787 11389 fs_manager.cc:359] Metadata directory not provided
03:07:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.735853 11389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.736027 11389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:07:05.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.738265 11389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/instance:
03:07:05.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:05.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.738848 11389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal/instance:
03:07:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.742754 11389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
03:07:05.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.745345 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:05.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.746711 11389 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.746874 11389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.746986 11389 fs_report.cc:383] FS layout report
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/data
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:05.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.756124 11389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:05.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.756798 11389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.757072 11389 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.757740 11389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.758665 11389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:07:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.758711 11389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
03:07:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.758750 11389 ts_tablet_manager.cc:610] Registered 0 tablets
03:07:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.758766 11389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
03:07:05.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.771838 11389 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.131:41163
03:07:05.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.771947 11535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.131:41163 every 8 connection(s)
03:07:05.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.772827 11389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:05.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.777577 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 11389
03:07:05.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.777707 10742 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal/instance
03:07:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.786501 11537 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.786552 11536 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.786787 11536 heartbeater.cc:464] Registering TS with master...
03:07:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.786789 11537 heartbeater.cc:464] Registering TS with master...
03:07:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.787196 11538 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.787269 11538 heartbeater.cc:464] Registering TS with master...
03:07:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.787402 11536 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.787406 11537 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.787413 11538 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.788596 10966 ts_manager.cc:194] Registered new tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.788602 10789 ts_manager.cc:194] Registered new tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.788602 10877 ts_manager.cc:194] Registered new tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:05.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.789382 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.131:53931
03:07:05.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.798575 10742 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
03:07:05.799 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
03:07:05.800 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

03:07:05.810 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.125.190"
      port: 42983
    }
    bound_http_address {
      host: "127.10.125.190"
      port: 40729
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.125.189"
      port: 33083
    }
    bound_http_address {
      host: "127.10.125.189"
      port: 43329
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.125.188"
      port: 32999
    }
    bound_http_address {
      host: "127.10.125.188"
      port: 32817
    }
  }
}

03:07:05.818 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

03:07:05.821 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.125.129"
      port: 46115
    }
    bound_http_address {
      host: "127.10.125.129"
      port: 44295
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.125.130"
      port: 32851
    }
    bound_http_address {
      host: "127.10.125.130"
      port: 42263
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.125.131"
      port: 41163
    }
    bound_http_address {
      host: "127.10.125.131"
      port: 38821
    }
  }
}

03:07:05.821 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
03:07:05.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.915555 11041 catalog_manager.cc:1259] Loaded cluster ID: 13eaa47b2e5645c29a9de1befe2915b5
03:07:05.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.915555 11055 catalog_manager.cc:1259] Loaded cluster ID: 13eaa47b2e5645c29a9de1befe2915b5
03:07:05.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.915639 11041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: loading cluster ID for follower catalog manager: success
03:07:05.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.915639 11055 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: loading cluster ID for follower catalog manager: success
03:07:05.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.918088 11055 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: acquiring CA information for follower catalog manager: success
03:07:05.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.918118 11041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: acquiring CA information for follower catalog manager: success
03:07:05.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.919628 11055 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 227faf0c829a4e83ac081987be322efd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
03:07:05.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:05.919628 11041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a7fb453b87e4166ae26f967cb15aab8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
03:07:06.120 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
03:07:06.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:06.342651 11226 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:06.381 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.125.189:33083(127.10.125.189:33083)] connecting to peer
03:07:06.448 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.125.190:42983(127.10.125.190:42983)] connecting to peer
03:07:06.450 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.125.188:32999(127.10.125.188:32999)] connecting to peer
03:07:06.490 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.10.125.190:42983(127.10.125.190:42983)] Successfully connected to peer
03:07:06.490 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.10.125.189:33083(127.10.125.189:33083)] Successfully connected to peer
03:07:06.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:06.572218 11382 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:06.574 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.10.125.188:32999(127.10.125.188:32999)] Successfully connected to peer
03:07:06.603 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.10.125.190
03:07:06.604 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.10.125.189
03:07:06.779 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.10.125.188
03:07:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:06.790968 11536 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:06.909 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x947e3594, L:/127.0.0.1:60734 - R:/127.10.125.190:42983] using SASL/PLAIN
03:07:06.909 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf88701bf, L:/127.0.0.1:34228 - R:/127.10.125.189:33083] using SASL/PLAIN
03:07:06.927 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x84d1746d, L:/127.0.0.1:32976 - R:/127.10.125.188:32999] using SASL/PLAIN
03:07:06.963 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
03:07:06.965 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=648), Traces: [0ms] refreshing cache from master, [45ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.125.189:33083, [121ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.125.190:42983, [122ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.125.188:32999, [597ms] Sub RPC ConnectToMaster: received response from server master-127.10.125.190:42983: OK, [597ms] Sub RPC ConnectToMaster: received response from server master-127.10.125.189:33083: OK, [605ms] Sub RPC ConnectToMaster: received response from server master-127.10.125.188:32999: OK, deferred=null) after lookup
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.028679 10966 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32976:
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297"
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
03:07:07.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
03:07:07.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:07.030112 10966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
03:07:07.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.053834 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 7c333092d11b4b969c4a8b742048aeaf (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297 [id=4d8362c8c7fa4e7c8111b4e84f2e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
03:07:07.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.054181 11462 tablet_service.cc:1467] Processing CreateTablet for tablet 7c333092d11b4b969c4a8b742048aeaf (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297 [id=4d8362c8c7fa4e7c8111b4e84f2e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
03:07:07.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.055259 11462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c333092d11b4b969c4a8b742048aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:07.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.055259 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c333092d11b4b969c4a8b742048aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:07.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.061802 11153 tablet_service.cc:1467] Processing CreateTablet for tablet 7c333092d11b4b969c4a8b742048aeaf (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297 [id=4d8362c8c7fa4e7c8111b4e84f2e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
03:07:07.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.062974 11153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c333092d11b4b969c4a8b742048aeaf. 1 dirs total, 0 dirs full, 0 dirs failed
03:07:07.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.063894 11573 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap starting.
03:07:07.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.065500 11574 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap starting.
03:07:07.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.066651 11573 tablet_bootstrap.cc:654] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Neither blocks nor log segments found. Creating new log.
03:07:07.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.068001 11573 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Log is configured to *not* fsync() on all Append() calls
03:07:07.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.068446 11574 tablet_bootstrap.cc:654] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Neither blocks nor log segments found. Creating new log.
03:07:07.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.069394 11574 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Log is configured to *not* fsync() on all Append() calls
03:07:07.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.070282 11573 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: No bootstrap required, opened a new log
03:07:07.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.070475 11573 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
03:07:07.074 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 4d8362c8c7fa4e7c8111b4e84f2e83b0
03:07:07.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.071520 11574 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: No bootstrap required, opened a new log
03:07:07.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.071743 11574 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
03:07:07.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.072404 11578 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap starting.
03:07:07.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.075409 11578 tablet_bootstrap.cc:654] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Neither blocks nor log segments found. Creating new log.
03:07:07.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.075937 11573 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.076453 11573 raft_consensus.cc:374] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:07.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.076578 11578 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Log is configured to *not* fsync() on all Append() calls
03:07:07.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.076647 11573 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Initialized, Role: FOLLOWER
03:07:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.076721 11574 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.077128 11574 raft_consensus.cc:374] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.077204 11574 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Initialized, Role: FOLLOWER
03:07:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.077221 11573 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.077658 11574 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.078336 11573 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
03:07:07.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.079111 11574 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
03:07:07.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.084869 11578 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: No bootstrap required, opened a new log
03:07:07.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.085047 11578 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.005s
03:07:07.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.089437 11578 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.089649 11578 raft_consensus.cc:374] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:07:07.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.089684 11578 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Initialized, Role: FOLLOWER
03:07:07.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.090055 11578 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.090962 11578 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
03:07:07.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.185739 11580 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:07:07.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.185953 11580 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.187366 11580 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:07.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.192801 11483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" is_pre_election: true
03:07:07.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.192883 11175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:07.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.193221 11175 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 0.
03:07:07.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.193222 11483 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 0.
03:07:07.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.193809 11257 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 
03:07:07.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.194108 11580 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Leader pre-election won for term 1
03:07:07.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.194211 11580 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
03:07:07.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.194273 11580 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 0 FOLLOWER]: Advancing to term 1
03:07:07.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.195395 11580 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.195820 11580 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 1 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:07.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.196215 11483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:07.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.196357 11175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:07.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.196364 11483 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 0 FOLLOWER]: Advancing to term 1
03:07:07.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.196463 11175 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 0 FOLLOWER]: Advancing to term 1
03:07:07.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.197541 11175 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 1.
03:07:07.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.197711 11483 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 1.
03:07:07.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.197888 11258 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: 
03:07:07.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.198154 11580 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Leader election won for term 1
03:07:07.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.198444 11580 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 LEADER]: Becoming Leader. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Running, Role: LEADER
03:07:07.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.198856 11580 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:07.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.202628 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d0e1e440184da1bb263d794b256d01 (127.10.125.130). New cstate: current_term: 1 leader_uuid: "30d0e1e440184da1bb263d794b256d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:07.226 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 4d8362c8c7fa4e7c8111b4e84f2e83b0
03:07:07.255 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:07.268 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299987, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:07.269 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=null) after lookup
03:07:07.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:07.275909 11539 tablet.cc:2369] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:07.282 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:07.284 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] Successfully connected to peer
03:07:07.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:07.306448 11385 tablet.cc:2369] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:07.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:07.322993 11229 tablet.cc:2369] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:07:07.363 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x953d3258, L:/127.0.0.1:35470 - R:/127.10.125.130:32851] using TOKEN/null
03:07:07.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.370906 11175 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Refusing update from remote peer 30d0e1e440184da1bb263d794b256d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:07:07.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.370908 11483 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Refusing update from remote peer 30d0e1e440184da1bb263d794b256d01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:07:07.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.371683 11585 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
03:07:07.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.371903 11580 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
03:07:07.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.382236 11587 mvcc.cc:205] Tried to move back new op lower bound from 7317520496103071744 to 7317520495409348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317520496103071744 or (T in {7317520496103071744})}]
03:07:07.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.383039 11588 mvcc.cc:205] Tried to move back new op lower bound from 7317520496103071744 to 7317520495409348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317520496103071744}]
03:07:07.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.386695 11589 mvcc.cc:205] Tried to move back new op lower bound from 7317520496103071744 to 7317520495409348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317520496103071744}]
03:07:07.482 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:07.485 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.130:32851
03:07:07.486 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:07.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:07.487028 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 11237
03:07:07.499 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:07.502 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] handling channelInactive
03:07:07.504 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state READY due to: connection closed
03:07:07.504 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: connection closed
03:07:07.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:07.522 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:07.523 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test)) after lookup
03:07:07.524 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:07.525 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.525 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.525 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.542 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:07.543 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:07.543 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185))) after lookup
03:07:07.544 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:07.545 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.545 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.545 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.563 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:07.563 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:07.564 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=64), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [46ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [47ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [63ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882))) after lookup
03:07:07.565 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:07.566 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.566 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.566 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.623 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:07.623 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:07.624 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=124), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [46ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [47ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [63ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [65ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [67ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [67ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [120ms] refreshing cache from master, [120ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [123ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824))) after lookup
03:07:07.625 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:07.627 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.627 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:07.627 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:08.082 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:08.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=583), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [46ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [47ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [63ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [65ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [67ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [67ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [120ms] refreshing cache from master, [120ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [123ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [126ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [128ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [128ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [579ms] refreshing cache from master, [580ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [582ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132))) after lookup
03:07:08.084 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:08.085 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.086 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.086 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:08.763 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:08.764 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1264), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [46ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [47ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [63ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [65ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [67ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [67ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [120ms] refreshing cache from master, [120ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [123ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [126ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [128ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [128ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [579ms] refreshing cache from master, [580ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [582ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [585ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [587ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [587ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1260ms] refreshing cache from master, [1260ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1263ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132) -> (continuation of Deferred@450626743 after retry RPC@2112349829), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132) -> (continuation of Deferred@450626743 after retry RPC@2112349829))) after lookup
03:07:08.765 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:08.766 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.767 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.767 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.922590 11599 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.922729 11599 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.923507 11600 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.923676 11600 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:08.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.924305 11599 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.925028 11600 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:08.925073 11098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:08.925937 11411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:08.926046 11098 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:08.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:08.926946 11411 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.929313 11483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" is_pre_election: true
03:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.929451 11483 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 1.
03:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.929808 11095 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.930056 11599 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Leader pre-election won for term 2
03:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.930125 11599 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.930167 11599 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 1 FOLLOWER]: Advancing to term 2
03:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.931516 11599 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.931888 11174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.931969 11599 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.932096 11174 raft_consensus.cc:2379] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd23193b2ab54c399c0ceb3bf17f7edd in current term 2: Already voted for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in this term.
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.932292 11483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.932437 11483 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 1 FOLLOWER]: Advancing to term 2
03:07:08.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.932530 11410 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01
03:07:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:08.933255 11098 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.933770 11483 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 2.
03:07:08.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.933931 11600 raft_consensus.cc:2726] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
03:07:08.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.934079 11095 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:08.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.934255 11599 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Leader election won for term 2
03:07:08.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.934497 11599 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 LEADER]: Becoming Leader. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Running, Role: LEADER
03:07:08.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.934846 11599 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:08.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:08.937853 10966 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 reported cstate change: term changed from 1 to 2, leader changed from 30d0e1e440184da1bb263d794b256d01 (127.10.125.130) to 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129). New cstate: current_term: 2 leader_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: HEALTHY } } }
03:07:09.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:09.413223 11098 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 -> Peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Couldn't send request to peer 30d0e1e440184da1bb263d794b256d01. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:09.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:09.513633 11483 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Refusing update from remote peer 1d45cdc346ff4fe4ad8e957a8ad1de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
03:07:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:09.514307 11599 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
03:07:11.283 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.283 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.284 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=3784), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [5ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [6ms] delaying RPC due to: Network error: connection closed, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [22ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [25ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [26ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [26ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [43ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [46ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [47ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [63ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [65ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [67ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [67ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [120ms] refreshing cache from master, [120ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [123ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [126ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [128ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [128ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [579ms] refreshing cache from master, [580ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [582ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [585ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [587ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [587ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1260ms] refreshing cache from master, [1260ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1263ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [1265ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1268ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1268ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3780ms] refreshing cache from master, [3780ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [3783ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1974907579(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@307f0bff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132) -> (continuation of Deferred@450626743 after retry RPC@2112349829) -> (continuation of Deferred@954100862 after retry RPC@2060416558), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6c05f4f9 -> wakeup thread Time-limited test -> (continuation of Deferred@1990400357 after retry RPC@256153185) -> (continuation of Deferred@780969020 after retry RPC@97572882) -> (continuation of Deferred@1854693145 after retry RPC@484582824) -> (continuation of Deferred@166614372 after retry RPC@78442132) -> (continuation of Deferred@450626743 after retry RPC@2112349829) -> (continuation of Deferred@954100862 after retry RPC@2060416558))) after lookup
03:07:11.286 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.287 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] Successfully connected to peer
03:07:11.305 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x3e8a317a, L:/127.0.0.1:37198 - R:/127.10.125.129:46115] using TOKEN/null
03:07:11.329 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.130:32851
03:07:11.331 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.331794 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:11.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.449138 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:11.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.449568 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:11.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.449627 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:11.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.455508 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:11.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.455631 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.130
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.462376 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11644
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.463776 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:11.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.465481 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:11.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.475597 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.475646 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:11.475651 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.475838 11644 server_base.cc:1034] running on GCE node
03:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.476263 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.476820 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:11.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.477982 11644 hybrid_clock.cc:648] HybridClock initialized: now 1786504031477951 us; error 51 us; skew 500 ppm
03:07:11.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.487013 11644 webserver.cc:460] Webserver started at http://127.10.125.130:42263/ using document root <none> and password file <none>
03:07:11.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.487690 11644 fs_manager.cc:359] Metadata directory not provided
03:07:11.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.487818 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:11.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.491743 11644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.004s	sys 0.000s
03:07:11.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.493872 11663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.495201 11644 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
03:07:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.495328 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.495950 11644 fs_report.cc:383] FS layout report
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/data
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:11.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:11.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.524729 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:11.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.525501 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:11.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.525722 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:11.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.526387 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:11.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.527733 11670 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:11.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.531622 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:11.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.531702 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:11.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.531770 11644 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.533380 11644 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.533452 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.533668 11670 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap starting.
03:07:11.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.548898 11644 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.130:32851
03:07:11.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.549016 11785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.130:32851 every 8 connection(s)
03:07:11.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.550015 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:11.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.559149 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 11644
03:07:11.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:11.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.561758 11670 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Log is configured to *not* fsync() on all Append() calls
03:07:11.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.564600 11787 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:11.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.564821 11787 heartbeater.cc:464] Registering TS with master...
03:07:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.565033 11789 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.565135 11789 heartbeater.cc:464] Registering TS with master...
03:07:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.565281 11787 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.565313 11789 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:11.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.567340 11788 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:11.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.567446 11788 heartbeater.cc:464] Registering TS with master...
03:07:11.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.567620 11788 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:11.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.568274 10965 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:11.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.568640 10789 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:11.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.569276 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.130:49365
03:07:11.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.570209 10877 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:11.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.573642 11670 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:11.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.573979 11670 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap complete.
03:07:11.575 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:11.576 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.129:46115
03:07:11.576 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:11.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.576721 11670 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent bootstrapping tablet: real 0.043s	user 0.026s	sys 0.011s
03:07:11.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.577899 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 11076
03:07:11.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.581858 11670 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:11.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.582283 11670 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Initialized, Role: FOLLOWER
03:07:11.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.582809 11670 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:11.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.583583 11787 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:11.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:11.584008 11670 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
03:07:11.596 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] handling channelInactive
03:07:11.596 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state READY due to: connection closed
03:07:11.596 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:11.597 [DEBUG - Time-limited test] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.598 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.598 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.599 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.622 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.623 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.624 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test)) after lookup
03:07:11.624 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.625 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.625 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.625 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.643 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.643 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635))) after lookup
03:07:11.645 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.646 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.646 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.647 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.682 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.682 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.683 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [50ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [51ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357))) after lookup
03:07:11.683 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.684 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.684 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.685 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.723 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=126), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [50ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [51ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [87ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [89ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637))) after lookup
03:07:11.725 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.727 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.727 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.728 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.943 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:11.943 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:11.944 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=347), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [50ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [51ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [87ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [89ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [128ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [131ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [132ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [343ms] refreshing cache from master, [343ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [346ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627))) after lookup
03:07:11.945 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:11.945 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.945 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:11.946 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:12.423 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:12.423 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:12.424 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=827), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [50ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [51ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [87ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [89ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [128ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [131ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [132ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [343ms] refreshing cache from master, [343ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [346ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [348ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [349ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [350ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [823ms] refreshing cache from master, [823ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [826ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627) -> (continuation of Deferred@1566427785 after retry RPC@2099567037), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627) -> (continuation of Deferred@1566427785 after retry RPC@2099567037))) after lookup
03:07:12.424 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:12.426 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:12.426 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:12.426 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:12.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.824564 11801 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:12.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.825322 11801 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:12.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.825935 11801 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:12.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:12.827394 11410 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:12.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.831595 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01" is_pre_election: true
03:07:12.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.831881 11738 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 1.
03:07:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.832669 11411 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.832847 11801 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Leader pre-election won for term 3
03:07:12.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.832912 11801 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:12.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.832967 11801 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 2 FOLLOWER]: Advancing to term 3
03:07:12.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.833870 11801 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:12.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.834210 11801 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 election: Requested vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:12.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.834553 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:12.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.834700 11738 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 1 FOLLOWER]: Advancing to term 3
03:07:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:12.835258 11410 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.836859 11738 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 3.
03:07:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.837251 11411 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.837455 11801 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Leader election won for term 3
03:07:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.837733 11801 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 LEADER]: Becoming Leader. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Running, Role: LEADER
03:07:12.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.838182 11801 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:12.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:12.840827 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd reported cstate change: term changed from 2 to 3, leader changed from 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129) to cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131). New cstate: current_term: 3 leader_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:13.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:13.354291 11738 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Refusing update from remote peer cd23193b2ab54c399c0ceb3bf17f7edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
03:07:13.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:13.355091 11821 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
03:07:13.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:13.365890 11410 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd -> Peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Couldn't send request to peer 1d45cdc346ff4fe4ad8e957a8ad1de47. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:13.940976 11410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111) [suppressed 3 similar messages]
03:07:14.442 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:14.443 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:14.444 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2847), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [3ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [29ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [29ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [45ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [50ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [51ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [82ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [87ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [89ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [128ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [131ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [132ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [343ms] refreshing cache from master, [343ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [346ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [348ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [349ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [350ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [823ms] refreshing cache from master, [823ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [826ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [828ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [830ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [830ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2843ms] refreshing cache from master, [2843ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [2845ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@105019763(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2672529 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627) -> (continuation of Deferred@1566427785 after retry RPC@2099567037) -> (continuation of Deferred@222497952 after retry RPC@1450300804), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cd62d64 -> wakeup thread Time-limited test -> (continuation of Deferred@657826008 after retry RPC@1836517635) -> (continuation of Deferred@790876691 after retry RPC@2051664357) -> (continuation of Deferred@696903198 after retry RPC@2094897637) -> (continuation of Deferred@553829675 after retry RPC@37517627) -> (continuation of Deferred@1566427785 after retry RPC@2099567037) -> (continuation of Deferred@222497952 after retry RPC@1450300804))) after lookup
03:07:14.445 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.446 [DEBUG - kudu-nio-0] (Connection.java:757) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] Successfully connected to peer
03:07:14.469 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x1d650d3c, L:/127.0.0.1:41428 - R:/127.10.125.131:41163] using TOKEN/null
03:07:14.490 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.129:46115
03:07:14.491 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.492250 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:14.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.612970 11849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:14.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.613232 11849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:14.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.613267 11849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.617280 11849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.617369 11849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.129
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.622143 11849 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:14.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11849
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.623212 11849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:14.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.624672 11849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:14.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.634655 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:14.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.634712 11849 server_base.cc:1034] running on GCE node
03:07:14.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.634663 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:14.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:14.634663 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:14.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.635654 11849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:14.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.636358 11849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:14.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.637557 11849 hybrid_clock.cc:648] HybridClock initialized: now 1786504034637518 us; error 47 us; skew 500 ppm
03:07:14.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.639671 11849 webserver.cc:460] Webserver started at http://127.10.125.129:44295/ using document root <none> and password file <none>
03:07:14.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.640281 11849 fs_manager.cc:359] Metadata directory not provided
03:07:14.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.640424 11849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:14.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.644281 11849 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.003s
03:07:14.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.646739 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:14.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.648132 11849 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:14.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.648278 11849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.648888 11849 fs_report.cc:383] FS layout report
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/data
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:14.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.663149 11849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.663882 11849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.664105 11849 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:14.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.664788 11849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:14.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.666145 11872 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.670324 11849 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.670383 11849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
03:07:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.670433 11849 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.672079 11849 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.672124 11849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
03:07:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.672519 11872 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap starting.
03:07:14.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.683248 11849 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.129:46115
03:07:14.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.683399 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.129:46115 every 8 connection(s)
03:07:14.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.684377 11849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:14.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.689625 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 11849
03:07:14.690 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:14.700 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:14.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.702266 11872 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Log is configured to *not* fsync() on all Append() calls
03:07:14.704 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.131:41163
03:07:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.704427 11997 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:14.704 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.704697 11997 heartbeater.cc:464] Registering TS with master...
03:07:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.705036 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 11389
03:07:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.705369 11997 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.708575 10966 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.709012 11999 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.709101 11999 heartbeater.cc:464] Registering TS with master...
03:07:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.709297 11999 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.709654 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.129:46059
03:07:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.710276 10878 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:14.715 [DEBUG - kudu-nio-0] (Connection.java:239) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] handling channelInactive
03:07:14.715 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state READY due to: connection closed
03:07:14.716 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:14.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.716548 11998 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:14.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.716655 11998 heartbeater.cc:464] Registering TS with master...
03:07:14.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.716871 11998 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:14.717 [DEBUG - Time-limited test] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.717782 10789 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:14.719 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.719 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.719 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.723708 11872 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.724086 11872 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap complete.
03:07:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.725509 11872 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent bootstrapping tablet: real 0.053s	user 0.016s	sys 0.019s
03:07:14.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.728092 11872 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:14.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.728520 11872 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Initialized, Role: FOLLOWER
03:07:14.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.729025 11872 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:14.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.729714 11997 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:14.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:14.730016 11872 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent starting tablet: real 0.004s	user 0.006s	sys 0.000s
03:07:14.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:14.742 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:14.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test)) after lookup
03:07:14.743 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.744 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.744 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.745 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:14.762 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:14.763 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [28ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898))) after lookup
03:07:14.763 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.765 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.765 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.766 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.802 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:14.802 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:14.803 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [28ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [46ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [49ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415))) after lookup
03:07:14.804 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.804 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.804 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.805 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.922 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:14.922 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:14.923 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=205), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [28ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [46ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [49ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [87ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [88ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [202ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [205ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992))) after lookup
03:07:14.924 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:14.926 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.926 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:14.926 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:15.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:15.082 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:15.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=365), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [28ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [46ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [49ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [87ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [88ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [202ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [205ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [207ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [209ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [209ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [362ms] refreshing cache from master, [362ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [365ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992) -> (continuation of Deferred@1968983193 after retry RPC@1731966579), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992) -> (continuation of Deferred@1968983193 after retry RPC@1731966579))) after lookup
03:07:15.084 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:15.084 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:15.084 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:15.085 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:15.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.989791 12011 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:15.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.989950 12011 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:15.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.991345 12011 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:15.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:15.992054 11665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:15.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:15.993012 11665 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:15.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.997315 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:15.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.997609 11943 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 2.
03:07:15.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.998343 11666 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:15.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.998590 12011 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Leader pre-election won for term 4
03:07:15.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.998653 12011 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:15.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.998694 12011 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 3 FOLLOWER]: Advancing to term 4
03:07:15.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:15.999804 12011 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:16.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.000146 12011 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:16.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.000542 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:16.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.000707 11943 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 2 FOLLOWER]: Advancing to term 4
03:07:16.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.001256 11665 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:16.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.003316 11943 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 4.
03:07:16.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.003652 11666 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:16.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.003876 12011 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Leader election won for term 4
03:07:16.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.004118 12011 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 LEADER]: Becoming Leader. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Running, Role: LEADER
03:07:16.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.004477 12011 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:16.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.007328 10966 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 reported cstate change: term changed from 3 to 4, leader changed from cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131) to 30d0e1e440184da1bb263d794b256d01 (127.10.125.130). New cstate: current_term: 4 leader_uuid: "30d0e1e440184da1bb263d794b256d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:16.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.478082 11665 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 -> Peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Couldn't send request to peer cd23193b2ab54c399c0ceb3bf17f7edd. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:16.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.581115 11943 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Refusing update from remote peer 30d0e1e440184da1bb263d794b256d01: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
03:07:16.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.581887 12011 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
03:07:16.662 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:16.662 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:16.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1945), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [21ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [28ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [46ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [49ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [87ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [88ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [202ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [205ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [207ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [209ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [209ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [362ms] refreshing cache from master, [362ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [365ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [366ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [368ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [368ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [1942ms] refreshing cache from master, [1942ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1945ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1966767345(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2bd2b91d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992) -> (continuation of Deferred@1968983193 after retry RPC@1731966579) -> (continuation of Deferred@1520737025 after retry RPC@614467732), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@607549b9 -> wakeup thread Time-limited test -> (continuation of Deferred@922369388 after retry RPC@2041165898) -> (continuation of Deferred@1098329411 after retry RPC@1965598415) -> (continuation of Deferred@1269221820 after retry RPC@1316716992) -> (continuation of Deferred@1968983193 after retry RPC@1731966579) -> (continuation of Deferred@1520737025 after retry RPC@614467732))) after lookup
03:07:16.664 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:16.665 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] Successfully connected to peer
03:07:16.689 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb2e40571, L:/127.0.0.1:52636 - R:/127.10.125.130:32851] using TOKEN/null
03:07:16.709 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.131:41163
03:07:16.709 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:16.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.709815 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:16.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.831159 12045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:16.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.831458 12045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:16.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.831496 12045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:16.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.835744 12045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:16.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.835830 12045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.131
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.840659 12045 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:16.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12045
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:16.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.841775 12045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:16.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.843127 12045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.853160 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.853168 12050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.853334 12045 server_base.cc:1034] running on GCE node
03:07:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:16.853216 12051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.853940 12045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.854627 12045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:16.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.855787 12045 hybrid_clock.cc:648] HybridClock initialized: now 1786504036855773 us; error 26 us; skew 500 ppm
03:07:16.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.861965 12045 webserver.cc:460] Webserver started at http://127.10.125.131:38821/ using document root <none> and password file <none>
03:07:16.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.862569 12045 fs_manager.cc:359] Metadata directory not provided
03:07:16.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.862653 12045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:16.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.866624 12045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.000s
03:07:16.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.868800 12060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:16.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.870467 12045 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.870612 12045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.871407 12045 fs_report.cc:383] FS layout report
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/data
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:16.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:16.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:16.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.884567 12045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:16.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.885192 12045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:16.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.885336 12045 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:16.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.885840 12045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:16.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.887045 12068 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:16.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.891458 12045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:16.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.891523 12045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
03:07:16.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.891570 12045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:16.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.893055 12045 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:16.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.893122 12045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:16.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.893392 12068 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap starting.
03:07:16.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.904125 12045 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.131:41163
03:07:16.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.905112 12045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:16.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.907338 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 12045
03:07:16.907 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:16.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.910362 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.131:41163 every 8 connection(s)
03:07:16.918 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:16.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.925665 12068 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Log is configured to *not* fsync() on all Append() calls
03:07:16.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.929440 12189 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:16.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.929721 12189 heartbeater.cc:464] Registering TS with master...
03:07:16.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.930377 12189 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:16.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.930917 12188 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:16.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.931025 12188 heartbeater.cc:464] Registering TS with master...
03:07:16.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.931186 12188 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.932415 10965 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.932495 10788 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.933432 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.131:56533
03:07:16.934 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.130:32851
03:07:16.934 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:16.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.935148 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 11644
03:07:16.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.943041 12190 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:16.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.943146 12190 heartbeater.cc:464] Registering TS with master...
03:07:16.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.943392 12190 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:16.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.944331 10878 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:16.947 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] handling channelInactive
03:07:16.948 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state READY due to: connection closed
03:07:16.949 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:16.949 [DEBUG - Time-limited test] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:16.951 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.951 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.952 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.953701 12068 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.954156 12068 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap complete.
03:07:16.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.955375 12068 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent bootstrapping tablet: real 0.062s	user 0.030s	sys 0.013s
03:07:16.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.957758 12068 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:16.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.958320 12068 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Initialized, Role: FOLLOWER
03:07:16.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.958951 12068 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:16.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.960209 12068 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
03:07:16.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:16.962 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:16.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test)) after lookup
03:07:16.963 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:16.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:16.960877 12188 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:16.965 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.965 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.965 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.981 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:16.981 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:16.982 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [14ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [16ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472))) after lookup
03:07:16.982 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:16.983 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.983 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:16.983 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.002 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:17.002 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:17.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [14ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [16ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [33ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [34ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [34ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [52ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204))) after lookup
03:07:17.004 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:17.006 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.006 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.006 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.041 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:17.042 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:17.042 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [14ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [16ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [33ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [34ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [34ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [52ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [55ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [57ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [57ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662))) after lookup
03:07:17.043 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:17.044 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.044 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.044 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.441 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:17.442 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:17.442 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=493), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [14ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [16ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [33ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [34ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [34ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [52ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [55ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [57ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [57ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [95ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [95ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [490ms] refreshing cache from master, [490ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [492ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662) -> (continuation of Deferred@427035512 after retry RPC@627114334), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662) -> (continuation of Deferred@427035512 after retry RPC@627114334))) after lookup
03:07:17.443 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:17.445 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.445 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:17.445 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:18.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.208592 12202 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:18.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.208742 12202 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:18.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.209815 12202 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:18.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:18.210634 11869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:18.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:18.211616 11869 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.215601 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" is_pre_election: true
03:07:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.215890 12135 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 3.
03:07:18.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.216674 11867 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:18.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.216924 12202 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Leader pre-election won for term 5
03:07:18.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.217038 12202 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:18.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.217092 12202 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 4 FOLLOWER]: Advancing to term 5
03:07:18.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.218178 12202 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:18.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.218639 12202 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:18.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.218953 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:18.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.219086 12135 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 3 FOLLOWER]: Advancing to term 5
03:07:18.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:18.219800 11869 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:18.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.221395 12135 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 5.
03:07:18.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.221704 11867 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:18.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.221895 12202 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Leader election won for term 5
03:07:18.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.222175 12202 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 LEADER]: Becoming Leader. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Running, Role: LEADER
03:07:18.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.222574 12202 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:18.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.225440 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 reported cstate change: term changed from 4 to 5, leader changed from 30d0e1e440184da1bb263d794b256d01 (127.10.125.130) to 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129). New cstate: current_term: 5 leader_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: HEALTHY } } }
03:07:18.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.615291 12135 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Refusing update from remote peer 1d45cdc346ff4fe4ad8e957a8ad1de47: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
03:07:18.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:18.616207 12202 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
03:07:18.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:18.624553 11869 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 -> Peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Couldn't send request to peer 30d0e1e440184da1bb263d794b256d01. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:19.382 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:19.382 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:19.383 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2433), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [14ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [16ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [16ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [33ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [34ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [34ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [52ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [55ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [57ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [57ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [89ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [95ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [95ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [490ms] refreshing cache from master, [490ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [492ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [494ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [496ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [496ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [2430ms] refreshing cache from master, [2430ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [2432ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1174359071(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@25241789 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662) -> (continuation of Deferred@427035512 after retry RPC@627114334) -> (continuation of Deferred@1991408153 after retry RPC@1534389856), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2c6363a7 -> wakeup thread Time-limited test -> (continuation of Deferred@963355577 after retry RPC@237146472) -> (continuation of Deferred@1470380836 after retry RPC@1899051204) -> (continuation of Deferred@102310481 after retry RPC@2115427662) -> (continuation of Deferred@427035512 after retry RPC@627114334) -> (continuation of Deferred@1991408153 after retry RPC@1534389856))) after lookup
03:07:19.383 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.384 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] Successfully connected to peer
03:07:19.406 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xc376c95e, L:/127.0.0.1:55298 - R:/127.10.125.129:46115] using TOKEN/null
03:07:19.425 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.130:32851
03:07:19.425 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.425809 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:19.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:19.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.547144 12235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:19.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.547415 12235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:19.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.547470 12235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:19.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.551350 12235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:19.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.551442 12235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.130
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.556169 12235 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12235
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:19.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:19.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:19.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:19.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:19.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.557199 12235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:19.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.558568 12235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:19.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.568621 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:19.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.568641 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:19.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:19.568627 12241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:19.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.568893 12235 server_base.cc:1034] running on GCE node
03:07:19.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.569434 12235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:19.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.570119 12235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:19.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.571413 12235 hybrid_clock.cc:648] HybridClock initialized: now 1786504039571389 us; error 41 us; skew 500 ppm
03:07:19.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.573482 12235 webserver.cc:460] Webserver started at http://127.10.125.130:42263/ using document root <none> and password file <none>
03:07:19.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.574156 12235 fs_manager.cc:359] Metadata directory not provided
03:07:19.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.574251 12235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:19.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.578238 12235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.002s
03:07:19.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.580505 12252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:19.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.581677 12235 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.581820 12235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.582386 12235 fs_report.cc:383] FS layout report
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/data
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:19.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:19.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:19.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.615947 12235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:19.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.616786 12235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:19.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.617024 12235 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:19.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.617688 12235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:19.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.619047 12259 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:19.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.623111 12235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:19.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.623171 12235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:19.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.623222 12235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:19.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.624848 12235 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:19.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.624922 12235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:19.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.625227 12259 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap starting.
03:07:19.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.636511 12235 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.130:32851
03:07:19.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.636572 12379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.130:32851 every 8 connection(s)
03:07:19.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.637650 12235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:19.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.643217 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 12235
03:07:19.643 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.644513 12259 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Log is configured to *not* fsync() on all Append() calls
03:07:19.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.651671 12380 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.651924 12380 heartbeater.cc:464] Registering TS with master...
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.652546 12380 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.654491 12381 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.654593 12381 heartbeater.cc:464] Registering TS with master...
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.654771 12381 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.654771 10965 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:19.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.655812 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.130:49927
03:07:19.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.655823 10788 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:19.659 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:19.660 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.129:46115
03:07:19.660 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:19.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.660673 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 11849
03:07:19.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.665017 12259 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:19.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.665481 12259 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap complete.
03:07:19.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.667057 12259 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent bootstrapping tablet: real 0.042s	user 0.023s	sys 0.012s
03:07:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.667729 12383 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.667827 12383 heartbeater.cc:464] Registering TS with master...
03:07:19.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.668071 12383 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:19.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.669054 10878 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:19.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.670439 12259 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:19.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.671128 12259 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Initialized, Role: FOLLOWER
03:07:19.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.671705 12259 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:19.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.672465 12380 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:19.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:19.672937 12259 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent starting tablet: real 0.006s	user 0.009s	sys 0.000s
03:07:19.677 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] handling channelInactive
03:07:19.677 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:19.678 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state READY due to: connection closed
03:07:19.678 [DEBUG - Time-limited test] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.680 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.680 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.680 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:19.703 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:19.703 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test)) after lookup
03:07:19.704 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.705 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.705 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.705 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:19.722 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:19.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [27ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [27ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634))) after lookup
03:07:19.723 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.725 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.725 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.725 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:19.762 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:19.763 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [27ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [27ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [47ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [84ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231))) after lookup
03:07:19.764 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.765 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.765 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.765 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.822 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:19.823 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:19.844 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=145), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [27ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [27ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [47ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [84ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [87ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [87ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [144ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493))) after lookup
03:07:19.846 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:19.848 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.848 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:19.849 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:20.362 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:20.363 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:20.363 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=685), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [27ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [27ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [47ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [84ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [87ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [87ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [144ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [167ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [171ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [171ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [681ms] refreshing cache from master, [681ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [684ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493) -> (continuation of Deferred@1176979242 after retry RPC@1943012779), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493) -> (continuation of Deferred@1176979242 after retry RPC@1943012779))) after lookup
03:07:20.364 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:20.365 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:20.365 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:20.365 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:20.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.924516 12395 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:20.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.924696 12395 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:20.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.925801 12395 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:20.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:20.926755 12063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:20.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:20.927647 12063 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:20.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.932282 12331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01" is_pre_election: true
03:07:20.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.932693 12331 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 4.
03:07:20.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.933840 12064 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:20.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.934165 12395 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Leader pre-election won for term 6
03:07:20.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.934226 12395 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:20.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.934260 12395 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 5 FOLLOWER]: Advancing to term 6
03:07:20.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.935222 12395 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:20.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.935649 12395 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 election: Requested vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:20.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.936046 12331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:20.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.936187 12331 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 4 FOLLOWER]: Advancing to term 6
03:07:20.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:20.936738 12063 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:20.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.938655 12331 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 6.
03:07:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.939019 12064 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.939220 12395 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Leader election won for term 6
03:07:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.939496 12395 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 LEADER]: Becoming Leader. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Running, Role: LEADER
03:07:20.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.939877 12395 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:20.942402 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd reported cstate change: term changed from 5 to 6, leader changed from 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129) to cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131). New cstate: current_term: 6 leader_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:21.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.400486 12331 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Refusing update from remote peer cd23193b2ab54c399c0ceb3bf17f7edd: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
03:07:21.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.401224 12395 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
03:07:21.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.410066 12063 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd -> Peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Couldn't send request to peer 1d45cdc346ff4fe4ad8e957a8ad1de47. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:21.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:21.643 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:21.643 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1965), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [2ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [24ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [26ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [27ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [27ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [45ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [47ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [47ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [84ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [86ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [87ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [87ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [141ms] refreshing cache from master, [141ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [144ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [167ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [171ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [171ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [681ms] refreshing cache from master, [681ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [684ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [686ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [687ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [687ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [1961ms] refreshing cache from master, [1961ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1964ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1586259119(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@4fa174ba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493) -> (continuation of Deferred@1176979242 after retry RPC@1943012779) -> (continuation of Deferred@108156443 after retry RPC@2117271535), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6d02283f -> wakeup thread Time-limited test -> (continuation of Deferred@566298807 after retry RPC@286413634) -> (continuation of Deferred@954878971 after retry RPC@1004640231) -> (continuation of Deferred@1320963856 after retry RPC@314892493) -> (continuation of Deferred@1176979242 after retry RPC@1943012779) -> (continuation of Deferred@108156443 after retry RPC@2117271535))) after lookup
03:07:21.644 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:21.645 [DEBUG - kudu-nio-1] (Connection.java:757) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] Successfully connected to peer
03:07:21.662 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x97a27c40, L:/127.0.0.1:41498 - R:/127.10.125.131:41163] using TOKEN/null
03:07:21.682 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.129:46115
03:07:21.683 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.684018 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:21.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.807134 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:21.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.807410 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:21.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.807524 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:21.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.811484 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:21.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.811566 12426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.129
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.816284 12426 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:21.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12426
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:21.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.817601 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:21.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.819481 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:21.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.830124 12431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:21.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.830175 12432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:21.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.830178 12435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:21.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.830297 12426 server_base.cc:1034] running on GCE node
03:07:21.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.830900 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:21.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.831664 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.832860 12426 hybrid_clock.cc:648] HybridClock initialized: now 1786504041832842 us; error 64 us; skew 500 ppm
03:07:21.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.834828 12426 webserver.cc:460] Webserver started at http://127.10.125.129:44295/ using document root <none> and password file <none>
03:07:21.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.835490 12426 fs_manager.cc:359] Metadata directory not provided
03:07:21.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.835610 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:21.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.839682 12426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
03:07:21.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.841895 12441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:21.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.843153 12426 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.843297 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.844035 12426 fs_report.cc:383] FS layout report
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/data
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:21.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.884258 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:21.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.885095 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:21.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.885316 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:21.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.886096 12426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:21.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.887593 12450 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:21.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.891741 12426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:21.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.891812 12426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:21.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.891877 12426 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:21.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.893476 12426 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:21.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.893535 12426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:21.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.893846 12450 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap starting.
03:07:21.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.905011 12426 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.129:46115
03:07:21.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.906219 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.908022 12573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.129:46115 every 8 connection(s)
03:07:21.912 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:21.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.911984 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 12426
03:07:21.918 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
03:07:21.919 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
03:07:21.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.923942 12577 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:21.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.924216 12577 heartbeater.cc:464] Registering TS with master...
03:07:21.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.924854 12577 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:21.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.926815 10878 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:21.930 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:21.931 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.131:41163
03:07:21.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.932137 12574 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:21.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.932237 12574 heartbeater.cc:464] Registering TS with master...
03:07:21.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.932435 12574 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:21.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.933390 12450 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Log is configured to *not* fsync() on all Append() calls
03:07:21.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.933518 10966 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:21.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.934141 12576 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:21.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.934216 12576 heartbeater.cc:464] Registering TS with master...
03:07:21.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.934360 12576 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:21.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.934506 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.129:38979
03:07:21.935 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:21.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.935312 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 12045
03:07:21.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.940029 10788 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:21.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:21.948521 12447 connection.cc:535] server connection from 127.10.125.131:35807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
03:07:21.949 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:21.949 [DEBUG - kudu-nio-1] (Connection.java:239) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] handling channelInactive
03:07:21.949 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state READY due to: connection closed
03:07:21.950 [DEBUG - Time-limited test] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:21.951 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.951 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.952 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.959581 12450 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:21.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.960140 12450 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap complete.
03:07:21.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.961581 12450 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent bootstrapping tablet: real 0.068s	user 0.020s	sys 0.027s
03:07:21.961 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:21.962 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:21.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test)) after lookup
03:07:21.963 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:21.964 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.964 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.965 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.964745 12450 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:21.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.965245 12450 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Initialized, Role: FOLLOWER
03:07:21.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.965982 12450 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:21.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.966869 12574 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:21.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:21.967178 12450 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
03:07:21.981 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:21.981 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:21.982 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454))) after lookup
03:07:21.983 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:21.984 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.984 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:21.984 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.001 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:22.001 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:22.002 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [34ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [35ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [35ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488))) after lookup
03:07:22.002 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:22.004 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.004 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.004 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:22.122 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:22.122 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=172), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [34ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [35ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [35ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [55ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [55ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [172ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914))) after lookup
03:07:22.124 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:22.124 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.124 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.125 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:22.522 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:22.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=572), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [34ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [35ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [35ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [55ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [55ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [172ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [174ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [176ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [176ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [570ms] refreshing cache from master, [570ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [572ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184))) after lookup
03:07:22.523 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:22.524 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.525 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.525 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.621 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:22.622 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:22.622 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=672), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [34ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [35ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [35ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [55ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [55ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [172ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [174ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [176ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [176ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [570ms] refreshing cache from master, [570ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [572ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [573ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [576ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [577ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [669ms] refreshing cache from master, [669ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [672ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184) -> (continuation of Deferred@1269538839 after retry RPC@50396123), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184) -> (continuation of Deferred@1269538839 after retry RPC@50396123))) after lookup
03:07:22.623 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:22.624 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.624 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:22.624 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.181520 12592 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.181654 12592 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:23.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.182785 12592 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:23.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:23.183511 12254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:23.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:23.184486 12254 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:23.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.187470 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:23.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.187842 12522 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 5.
03:07:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.188328 12255 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.188627 12592 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Leader pre-election won for term 7
03:07:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.188686 12592 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.188715 12592 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 6 FOLLOWER]: Advancing to term 7
03:07:23.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.189687 12592 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:23.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.190052 12592 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:23.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.190486 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:23.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.190626 12522 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 5 FOLLOWER]: Advancing to term 7
03:07:23.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:23.190975 12254 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:23.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.192906 12522 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 7.
03:07:23.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.193253 12255 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:23.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.193466 12592 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Leader election won for term 7
03:07:23.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.193656 12592 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 LEADER]: Becoming Leader. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Running, Role: LEADER
03:07:23.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.193969 12592 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:23.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.196411 10966 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 reported cstate change: term changed from 6 to 7, leader changed from cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131) to 30d0e1e440184da1bb263d794b256d01 (127.10.125.130). New cstate: current_term: 7 leader_uuid: "30d0e1e440184da1bb263d794b256d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:23.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:23.599674 12254 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 -> Peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Couldn't send request to peer cd23193b2ab54c399c0ceb3bf17f7edd. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:23.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.744542 12522 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Refusing update from remote peer 30d0e1e440184da1bb263d794b256d01: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
03:07:23.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:23.745299 12592 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
03:07:25.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:25.994165 12254 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 -> Peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Couldn't send request to peer cd23193b2ab54c399c0ceb3bf17f7edd. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
03:07:26.141 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.141 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.142 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4192), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [3ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [13ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [16ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [16ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [34ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [35ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [35ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [55ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [55ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [172ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [174ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [176ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [176ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [570ms] refreshing cache from master, [570ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [572ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [573ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [576ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [577ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [669ms] refreshing cache from master, [669ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [672ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [673ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [675ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [675ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [4189ms] refreshing cache from master, [4189ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [4192ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1940619287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@342518f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184) -> (continuation of Deferred@1269538839 after retry RPC@50396123) -> (continuation of Deferred@1263949162 after retry RPC@1555962554), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5547892e -> wakeup thread Time-limited test -> (continuation of Deferred@1276075168 after retry RPC@627614454) -> (continuation of Deferred@2005707968 after retry RPC@1465442488) -> (continuation of Deferred@1590939472 after retry RPC@1319024914) -> (continuation of Deferred@1307358710 after retry RPC@871330184) -> (continuation of Deferred@1269538839 after retry RPC@50396123) -> (continuation of Deferred@1263949162 after retry RPC@1555962554))) after lookup
03:07:26.143 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.145 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] Successfully connected to peer
03:07:26.162 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x3fe7ccf0, L:/127.0.0.1:58304 - R:/127.10.125.130:32851] using TOKEN/null
03:07:26.180 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.131:41163
03:07:26.180 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.180994 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:26.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:26.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.316883 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:26.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.317168 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:26.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.317204 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.321353 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.321453 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.131
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.326153 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12638
03:07:26.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:26.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.327368 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:26.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.328825 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:26.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.338549 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:26.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.338577 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:26.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.338660 12638 server_base.cc:1034] running on GCE node
03:07:26.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.338585 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:26.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.339282 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:26.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.339936 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:26.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.341126 12638 hybrid_clock.cc:648] HybridClock initialized: now 1786504046341103 us; error 32 us; skew 500 ppm
03:07:26.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.342926 12638 webserver.cc:460] Webserver started at http://127.10.125.131:38821/ using document root <none> and password file <none>
03:07:26.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.343600 12638 fs_manager.cc:359] Metadata directory not provided
03:07:26.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.343694 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:26.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.347572 12638 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.004s	sys 0.000s
03:07:26.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.349748 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.350963 12638 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.351094 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.351727 12638 fs_report.cc:383] FS layout report
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:26.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/data
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:26.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:26.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.376610 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:26.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.377432 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:26.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.377650 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:26.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.378346 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:26.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.379719 12662 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:26.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.383773 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:26.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.383848 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:26.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.383916 12638 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:26.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.385576 12638 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:26.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.385643 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:26.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.385885 12662 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap starting.
03:07:26.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.396874 12638 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.131:41163
03:07:26.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.396967 12788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.131:41163 every 8 connection(s)
03:07:26.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.397934 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:26.399 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.398243 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 12638
03:07:26.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.408460 12662 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Log is configured to *not* fsync() on all Append() calls
03:07:26.409 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:26.411 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.130:32851
03:07:26.411 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:26.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.411679 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 12235
03:07:26.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.413245 12791 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:26.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.413502 12791 heartbeater.cc:464] Registering TS with master...
03:07:26.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.414163 12791 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:26.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.416234 10878 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:26.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.419014 12789 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:26.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.419103 12789 heartbeater.cc:464] Registering TS with master...
03:07:26.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.419310 12789 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:26.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.420130 12790 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:26.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.420209 12790 heartbeater.cc:464] Registering TS with master...
03:07:26.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.420382 12790 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:26.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.420526 10966 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:26.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.421681 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.131:50641
03:07:26.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.422384 10788 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:26.424283 12796 negotiation.cc:337] Failed RPC negotiation. Trace:
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.402241 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.125.130:34665 (local address 127.10.125.131:41163)
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404396 (+  2155us) server_negotiation.cc:206] Beginning negotiation
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404423 (+    27us) server_negotiation.cc:399] Waiting for connection header
03:07:26.424 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] handling channelInactive
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404458 (+    35us) server_negotiation.cc:407] Connection header received
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404582 (+   124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404588 (+     6us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.404991 (+   403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.406500 (+  1509us) server_negotiation.cc:226] Negotiated authn=SASL
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.406980 (+   480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:07:26.424 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state READY due to: connection closed
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.408858 (+  1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.410323 (+  1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.414348 (+  4025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:07:26.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.414454 (+   106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
03:07:26.425 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:26.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 03:07:26.423240 (+  8786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.125.130:34665: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
03:07:26.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":3204,"server-negotiator.queue_time_us":2018,"thread_start_us":165,"threads_started":1}
03:07:26.426 [DEBUG - Time-limited test] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.427 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.427 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.427 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.433166 12662 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:26.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.433621 12662 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap complete.
03:07:26.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.434866 12662 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent bootstrapping tablet: real 0.049s	user 0.031s	sys 0.007s
03:07:26.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.437431 12662 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:26.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.437975 12662 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Initialized, Role: FOLLOWER
03:07:26.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.438477 12662 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:26.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.439273 12789 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:26.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:26.439615 12662 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
03:07:26.442 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.443 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.443 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test)) after lookup
03:07:26.444 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.445 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.445 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.446 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.461 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.461 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.462 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652))) after lookup
03:07:26.463 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.464 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.464 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.464 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.503 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.503 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [37ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [39ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [39ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [77ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040))) after lookup
03:07:26.504 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.505 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.505 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.506 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.581 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.582 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.582 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=156), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [37ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [39ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [39ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [77ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [78ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [81ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [81ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [156ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577))) after lookup
03:07:26.583 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.584 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.584 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.584 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:26.762 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:26.762 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=337), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [37ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [39ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [39ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [77ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [78ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [81ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [81ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [156ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [158ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [159ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [159ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [334ms] refreshing cache from master, [334ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [336ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750))) after lookup
03:07:26.763 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:26.765 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.765 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:26.765 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:27.621 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:27.621 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:27.622 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1196), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [37ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [39ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [39ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [77ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [78ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [81ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [81ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [156ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [158ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [159ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [159ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [334ms] refreshing cache from master, [334ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [336ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [338ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [340ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [340ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1193ms] refreshing cache from master, [1194ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1196ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750) -> (continuation of Deferred@2106233072 after retry RPC@79308196), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750) -> (continuation of Deferred@2106233072 after retry RPC@79308196))) after lookup
03:07:27.622 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:27.623 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:27.623 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:27.624 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:27.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.665959 12806 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:07:27.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.666157 12806 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:27.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.667349 12806 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:27.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:27.668177 12658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:27.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:27.669071 12658 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:27.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.671871 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:27.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.672466 12657 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01
03:07:27.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.672709 12806 raft_consensus.cc:2726] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
03:07:27.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.680286 12811 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:27.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.680387 12811 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:27.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.681391 12811 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:27.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:27.682112 12447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:27.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:27.683020 12447 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.686025 12732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" is_pre_election: true
03:07:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.686257 12732 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 6.
03:07:27.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.687049 12444 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:27.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.687337 12811 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Leader pre-election won for term 8
03:07:27.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.687415 12811 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:27.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.687480 12811 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 7 FOLLOWER]: Advancing to term 8
03:07:27.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.688481 12811 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:27.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.688851 12811 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:27.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.689196 12732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:27.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.689337 12732 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 6 FOLLOWER]: Advancing to term 8
03:07:27.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:27.689942 12447 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:27.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.691596 12732 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 8.
03:07:27.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.691872 12444 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:27.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.692039 12811 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Leader election won for term 8
03:07:27.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.692236 12811 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 LEADER]: Becoming Leader. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Running, Role: LEADER
03:07:27.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.692543 12811 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:27.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:27.695044 10966 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 reported cstate change: term changed from 7 to 8, leader changed from 30d0e1e440184da1bb263d794b256d01 (127.10.125.130) to 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129). New cstate: current_term: 8 leader_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: HEALTHY } } }
03:07:28.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:28.095011 12732 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Refusing update from remote peer 1d45cdc346ff4fe4ad8e957a8ad1de47: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
03:07:28.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:28.095786 12811 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
03:07:28.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:28.105304 12447 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 -> Peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Couldn't send request to peer 30d0e1e440184da1bb263d794b256d01. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:30.402 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:30.403 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:30.403 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3978), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [2ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [18ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [21ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [21ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [37ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [39ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [39ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [77ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [78ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [81ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [81ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [156ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [158ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [159ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [159ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [334ms] refreshing cache from master, [334ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [336ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [338ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [340ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [340ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1193ms] refreshing cache from master, [1194ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1196ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [1197ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1199ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [1199ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [3974ms] refreshing cache from master, [3974ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [3976ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1502717451(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@51b8a4ab -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750) -> (continuation of Deferred@2106233072 after retry RPC@79308196) -> (continuation of Deferred@1529058462 after retry RPC@1330697614), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7070eec -> wakeup thread Time-limited test -> (continuation of Deferred@15691683 after retry RPC@1059086652) -> (continuation of Deferred@1271383409 after retry RPC@751783040) -> (continuation of Deferred@1219310880 after retry RPC@508611577) -> (continuation of Deferred@994190032 after retry RPC@2010239750) -> (continuation of Deferred@2106233072 after retry RPC@79308196) -> (continuation of Deferred@1529058462 after retry RPC@1330697614))) after lookup
03:07:30.405 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:30.406 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] Successfully connected to peer
03:07:30.423 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x94f002d8, L:/127.0.0.1:32772 - R:/127.10.125.129:46115] using TOKEN/null
03:07:30.442 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.130:32851
03:07:30.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.443310 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:30.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.579579 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:30.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.579900 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:30.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.579964 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:30.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.580736 12447 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 -> Peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Couldn't send request to peer 30d0e1e440184da1bb263d794b256d01. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
03:07:30.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.584192 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:30.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.584314 12852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.130
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.589161 12852 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12852
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.590547 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:30.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.592048 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:30.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.601848 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:30.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.601825 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:30.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:30.601861 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:30.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.602073 12852 server_base.cc:1034] running on GCE node
03:07:30.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.602770 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:30.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.603494 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:30.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.604671 12852 hybrid_clock.cc:648] HybridClock initialized: now 1786504050604638 us; error 52 us; skew 500 ppm
03:07:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.610159 12852 webserver.cc:460] Webserver started at http://127.10.125.130:42263/ using document root <none> and password file <none>
03:07:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.610710 12852 fs_manager.cc:359] Metadata directory not provided
03:07:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.610773 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:30.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.614552 12852 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
03:07:30.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.616717 12871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:30.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.617918 12852 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
03:07:30.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.618151 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:30.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:30.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.618801 12852 fs_report.cc:383] FS layout report
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/data
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:30.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.635247 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:30.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.636005 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:30.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.636224 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:30.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.636893 12852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:30.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.638267 12880 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:30.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.642311 12852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:30.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.642390 12852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
03:07:30.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.642464 12852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:30.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.644519 12852 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:30.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.644583 12852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
03:07:30.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.644876 12880 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap starting.
03:07:30.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.655781 12852 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.130:32851
03:07:30.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.655860 13006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.130:32851 every 8 connection(s)
03:07:30.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.656888 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:30.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.662057 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 12852
03:07:30.662 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:30.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.665547 12880 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Log is configured to *not* fsync() on all Append() calls
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.671839 13009 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.672101 13009 heartbeater.cc:464] Registering TS with master...
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.672763 13009 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.672834 13008 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.672899 13008 heartbeater.cc:464] Registering TS with master...
03:07:30.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.673079 13008 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:30.675 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:30.676 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.129:46115
03:07:30.676 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.674981 10878 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.675107 10788 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.676358 13007 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.676431 13007 heartbeater.cc:464] Registering TS with master...
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.676559 13007 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.677088 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 12426
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.681200 10965 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.682377 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.130:56693
03:07:30.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.688733 12880 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:30.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.689214 12880 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap complete.
03:07:30.690 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:30.690 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] handling channelInactive
03:07:30.690 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state READY due to: connection closed
03:07:30.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.690709 12880 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent bootstrapping tablet: real 0.046s	user 0.039s	sys 0.000s
03:07:30.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.693154 12880 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:30.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.693606 12880 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Initialized, Role: FOLLOWER
03:07:30.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.694118 12880 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:30.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.694833 13007 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:30.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:30.695240 12880 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Time spent starting tablet: real 0.004s	user 0.007s	sys 0.000s
03:07:30.699 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:30.701 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.701 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.702 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.721 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:30.721 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:30.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=3, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test)) after lookup
03:07:30.722 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:30.724 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.724 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.724 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:30.742 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:30.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=5, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158))) after lookup
03:07:30.743 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:30.744 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.744 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.744 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.781 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:30.782 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:30.782 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=7, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=92), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [54ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [91ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602))) after lookup
03:07:30.783 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:30.785 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.785 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:30.785 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.042 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:31.042 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:31.042 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=9, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=352), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [54ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [91ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [95ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [95ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [351ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708))) after lookup
03:07:31.043 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:31.044 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.044 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.044 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.321 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:31.321 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:31.322 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=11, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=631), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [54ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [91ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [95ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [95ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [351ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [353ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [354ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [354ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [628ms] refreshing cache from master, [628ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [631ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028))) after lookup
03:07:31.323 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:31.325 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.325 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.325 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.942452 13022 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:31.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.942606 13022 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:31.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.943043 13022 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:31.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:31.944496 12657 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:31.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.948804 12953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01" is_pre_election: true
03:07:31.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.949095 12953 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 7.
03:07:31.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.949831 12658 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:31.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.950047 13022 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Leader pre-election won for term 9
03:07:31.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.950121 13022 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1d45cdc346ff4fe4ad8e957a8ad1de47)
03:07:31.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.950165 13022 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 8 FOLLOWER]: Advancing to term 9
03:07:31.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.951208 13022 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:31.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.951644 13022 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 election: Requested vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.951988 12953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.952119 12953 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 7 FOLLOWER]: Advancing to term 9
03:07:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:31.952744 12657 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111)
03:07:31.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.954381 12953 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd23193b2ab54c399c0ceb3bf17f7edd in term 9.
03:07:31.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.954684 12658 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30d0e1e440184da1bb263d794b256d01, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47
03:07:31.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.954852 13022 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Leader election won for term 9
03:07:31.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.955123 13022 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 LEADER]: Becoming Leader. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Running, Role: LEADER
03:07:31.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.955582 13022 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:31.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:31.958145 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd reported cstate change: term changed from 8 to 9, leader changed from 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129) to cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131). New cstate: current_term: 9 leader_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:31.961 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:31.961 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:31.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=13, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1272), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [54ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [91ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [95ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [95ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [351ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [353ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [354ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [354ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [628ms] refreshing cache from master, [628ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [631ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [632ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [635ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [635ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [1268ms] refreshing cache from master, [1268ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1271ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028) -> (continuation of Deferred@154709359 after retry RPC@1675899191), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028) -> (continuation of Deferred@154709359 after retry RPC@1675899191))) after lookup
03:07:31.965 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:31.965 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.965 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state CONNECTING due to: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:31.966 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115
03:07:32.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:32.393695 12657 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd -> Peer 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115): Couldn't send request to peer 1d45cdc346ff4fe4ad8e957a8ad1de47. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:32.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:32.463577 12953 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Refusing update from remote peer cd23193b2ab54c399c0ceb3bf17f7edd: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
03:07:32.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:32.464391 13022 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
03:07:32.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:32.469149 13046 mvcc.cc:205] Tried to move back new op lower bound from 7317520590575955968 to 7317520579351293952. Current Snapshot: MvccSnapshot[applied={T|T < 7317520590549385216}]
03:07:32.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:32.886009 12657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.129:46115: connect: Connection refused (error 111) [suppressed 3 similar messages]
03:07:32.922 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:32.922 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:32.922 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=15, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2232), Traces: [0ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [0ms] delaying RPC due to: Illegal state: connection is terminated, [9ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [11ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [12ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [34ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [34ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [53ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [54ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [54ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [91ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [93ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [95ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [95ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [349ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [351ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [353ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [354ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [354ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [628ms] refreshing cache from master, [628ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [631ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [632ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [635ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [635ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [1268ms] refreshing cache from master, [1268ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1271ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [1274ms] sending RPC to server 1d45cdc346ff4fe4ad8e957a8ad1de47, [1276ms] received response from server 1d45cdc346ff4fe4ad8e957a8ad1de47: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [1276ms] delaying RPC due to: Network error: Failed to connect to peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115): Connection refused: /127.10.125.129:46115, [2229ms] refreshing cache from master, [2229ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [2231ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@616142257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@2ada7f06 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028) -> (continuation of Deferred@154709359 after retry RPC@1675899191) -> (continuation of Deferred@1205082402 after retry RPC@769597008), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@64a8d753 -> wakeup thread Time-limited test -> (continuation of Deferred@1487618783 after retry RPC@2048595158) -> (continuation of Deferred@841277379 after retry RPC@485248602) -> (continuation of Deferred@1381806763 after retry RPC@742172708) -> (continuation of Deferred@227283129 after retry RPC@174857028) -> (continuation of Deferred@154709359 after retry RPC@1675899191) -> (continuation of Deferred@1205082402 after retry RPC@769597008))) after lookup
03:07:32.923 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:32.925 [DEBUG - kudu-nio-1] (Connection.java:757) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] Successfully connected to peer
03:07:32.940 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x9e1f9554, L:/127.0.0.1:58778 - R:/127.10.125.131:41163] using TOKEN/null
03:07:32.956 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.129:46115
03:07:32.957 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:32.957540 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:33.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.074056 13058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:33.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.074295 13058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:33.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.074329 13058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:33.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.078298 13058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:33.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.078387 13058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.129
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.082914 13058 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.129:46115
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.129
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44295
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.13058
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.129
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/logs
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:33.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.084095 13058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:33.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.085548 13058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:33.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.095389 13064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:33.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.095541 13068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:33.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:33.095446 13066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:33.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.095435 13058 server_base.cc:1034] running on GCE node
03:07:33.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.096125 13058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:33.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.096726 13058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:33.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.097920 13058 hybrid_clock.cc:648] HybridClock initialized: now 1786504053097887 us; error 42 us; skew 500 ppm
03:07:33.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.099870 13058 webserver.cc:460] Webserver started at http://127.10.125.129:44295/ using document root <none> and password file <none>
03:07:33.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.100474 13058 fs_manager.cc:359] Metadata directory not provided
03:07:33.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.100567 13058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:33.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.104849 13058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
03:07:33.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.107054 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:33.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.108332 13058 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.108482 13058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.109032 13058 fs_report.cc:383] FS layout report
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/wal
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/data
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:33.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:33.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.123261 13058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:33.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.124012 13058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:33.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.124243 13058 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:33.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.124934 13058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:33.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.126200 13083 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:33.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.130127 13058 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:33.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.130206 13058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:33.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.130272 13058 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.131949 13058 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.132014 13058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
03:07:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.132282 13083 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap starting.
03:07:33.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.142989 13058 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.129:46115
03:07:33.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.143110 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.129:46115 every 8 connection(s)
03:07:33.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.144090 13058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-0/data/info.pb
03:07:33.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.144501 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 13058
03:07:33.144 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:33.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.152042 13083 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Log is configured to *not* fsync() on all Append() calls
03:07:33.153 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:33.156 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.131:41163
03:07:33.157 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:33.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.157423 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 12638
03:07:33.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.162446 13211 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:33.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.162732 13211 heartbeater.cc:464] Registering TS with master...
03:07:33.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.163353 13211 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:33.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.163698 13209 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:33.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.163792 13209 heartbeater.cc:464] Registering TS with master...
03:07:33.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.163978 13209 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:33.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.162482 13210 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:33.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.164729 13210 heartbeater.cc:464] Registering TS with master...
03:07:33.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.164925 13210 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:33.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.165343 10965 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:33.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.165860 10877 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:33.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.166503 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.129:45875
03:07:33.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.167142 10788 ts_manager.cc:194] Re-registered known tserver with Master: 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:33.171 [DEBUG - kudu-nio-1] (Connection.java:239) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] handling channelInactive
03:07:33.171 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:33.171 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state READY due to: connection closed
03:07:33.172 [DEBUG - Time-limited test] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.173 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.173 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.173 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.181 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:33.181 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:33.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.181434 13083 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:33.181 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test)) after lookup
03:07:33.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.181958 13083 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Bootstrap complete.
03:07:33.182 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.183552 13083 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent bootstrapping tablet: real 0.051s	user 0.036s	sys 0.008s
03:07:33.183 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.183 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.184 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.186344 13083 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:33.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.186760 13083 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Initialized, Role: FOLLOWER
03:07:33.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.187232 13083 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:33.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.188117 13209 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:33.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:33.188313 13083 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
03:07:33.202 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:33.202 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:33.203 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844))) after lookup
03:07:33.203 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.204 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.204 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.204 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.241 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:33.242 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:33.242 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [33ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [33ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332))) after lookup
03:07:33.242 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.243 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.244 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.244 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.341 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:33.341 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:33.342 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=170), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [33ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [33ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [71ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [73ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [73ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [168ms] refreshing cache from master, [168ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [170ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858))) after lookup
03:07:33.342 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.343 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.343 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.343 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:33.742 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:33.742 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=571), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [33ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [33ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [71ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [73ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [73ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [168ms] refreshing cache from master, [168ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [170ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [171ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [172ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [172ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [568ms] refreshing cache from master, [568ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [570ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725))) after lookup
03:07:33.743 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:33.744 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.744 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:33.744 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:34.321 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:34.322 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)[L]]}]
03:07:34.322 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1150), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [33ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [33ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [71ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [73ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [73ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [168ms] refreshing cache from master, [168ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [170ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [171ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [172ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [172ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [568ms] refreshing cache from master, [568ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [570ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [572ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [573ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [574ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [1147ms] refreshing cache from master, [1147ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1150ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725) -> (continuation of Deferred@329089391 after retry RPC@1316998632), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725) -> (continuation of Deferred@329089391 after retry RPC@1316998632))) after lookup
03:07:34.323 [DEBUG - kudu-nio-0] (Connection.java:748) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] connecting to peer
03:07:34.324 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:34.324 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] cleaning up while in state CONNECTING due to: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:34.324 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163
03:07:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.457175 13225 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:34.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.457324 13225 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:34.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.458524 13225 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:34.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:34.459519 12874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:34.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:34.460417 12874 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:34.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.465185 13154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:34.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.465528 13154 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 8.
03:07:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.466382 12875 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.466645 13225 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Leader pre-election won for term 10
03:07:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.466706 13225 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cd23193b2ab54c399c0ceb3bf17f7edd)
03:07:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.466799 13225 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 9 FOLLOWER]: Advancing to term 10
03:07:34.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.467830 13225 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:34.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.468152 13225 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:34.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.468475 13154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "30d0e1e440184da1bb263d794b256d01" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47"
03:07:34.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.468634 13154 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 8 FOLLOWER]: Advancing to term 10
03:07:34.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:34.469077 12874 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111)
03:07:34.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.470830 13154 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d0e1e440184da1bb263d794b256d01 in term 10.
03:07:34.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.471151 12875 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01; no voters: cd23193b2ab54c399c0ceb3bf17f7edd
03:07:34.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.471310 13225 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 10 FOLLOWER]: Leader election won for term 10
03:07:34.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.471568 13225 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [term 10 LEADER]: Becoming Leader. State: Replica: 30d0e1e440184da1bb263d794b256d01, State: Running, Role: LEADER
03:07:34.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.471868 13225 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:34.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.474275 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 reported cstate change: term changed from 9 to 10, leader changed from cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131) to 30d0e1e440184da1bb263d794b256d01 (127.10.125.130). New cstate: current_term: 10 leader_uuid: "30d0e1e440184da1bb263d794b256d01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: UNKNOWN } } }
03:07:34.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.935117 13154 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Refusing update from remote peer 30d0e1e440184da1bb263d794b256d01: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
03:07:34.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:34.935945 13225 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
03:07:34.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:34.945355 12874 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01 -> Peer cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163): Couldn't send request to peer cd23193b2ab54c399c0ceb3bf17f7edd. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.131:41163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:35.901 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:35.902 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:35.902 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2730), Traces: [0ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [2ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [10ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [13ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [13ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [32ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [33ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [33ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [71ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [73ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [73ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [168ms] refreshing cache from master, [168ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [170ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [171ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [172ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [172ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [568ms] refreshing cache from master, [568ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [570ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [572ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [573ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [574ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [1147ms] refreshing cache from master, [1147ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [1150ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [1152ms] sending RPC to server cd23193b2ab54c399c0ceb3bf17f7edd, [1153ms] received response from server cd23193b2ab54c399c0ceb3bf17f7edd: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [1153ms] delaying RPC due to: Network error: Failed to connect to peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163): Connection refused: /127.10.125.131:41163, [2728ms] refreshing cache from master, [2728ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [2730ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1685520240(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@10d36713 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725) -> (continuation of Deferred@329089391 after retry RPC@1316998632) -> (continuation of Deferred@1137904323 after retry RPC@507617511), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@538a1ce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1491918739 after retry RPC@1526891844) -> (continuation of Deferred@151348175 after retry RPC@919306332) -> (continuation of Deferred@1732805343 after retry RPC@233518858) -> (continuation of Deferred@456184709 after retry RPC@1668798725) -> (continuation of Deferred@329089391 after retry RPC@1316998632) -> (continuation of Deferred@1137904323 after retry RPC@507617511))) after lookup
03:07:35.903 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:35.904 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] Successfully connected to peer
03:07:35.920 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x225b45e0, L:/127.0.0.1:49298 - R:/127.10.125.130:32851] using TOKEN/null
03:07:35.937 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.131:41163
03:07:35.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:35.938184 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:35.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:35.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:36.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.057658 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:36.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.057934 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:36.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.057971 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:36.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.062273 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:36.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.062358 13260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.131
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.067189 13260 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.131:41163
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.131
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38821
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.13260
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.131
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/logs
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:36.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:36.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.068271 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:36.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.069727 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:36.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.079480 13268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:36.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.079488 13270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:36.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.079677 13260 server_base.cc:1034] running on GCE node
03:07:36.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:36.079411 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:36.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.080277 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:36.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.080881 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:36.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.082074 13260 hybrid_clock.cc:648] HybridClock initialized: now 1786504056082062 us; error 23 us; skew 500 ppm
03:07:36.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.090696 13260 webserver.cc:460] Webserver started at http://127.10.125.131:38821/ using document root <none> and password file <none>
03:07:36.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.091370 13260 fs_manager.cc:359] Metadata directory not provided
03:07:36.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.091495 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:36.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.095304 13260 fs_manager.cc:709] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
03:07:36.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.097750 13276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:36.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.099128 13260 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.099270 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.099973 13260 fs_report.cc:383] FS layout report
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/wal
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/data
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:36.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:36.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.118084 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:36.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.118932 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:36.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.119161 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:36.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.119905 13260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:36.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.121277 13285 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:36.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.125500 13260 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:36.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.125578 13260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s	user 0.000s	sys 0.000s
03:07:36.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.125646 13260 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:36.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.127221 13260 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:36.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.127291 13260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:36.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.127665 13285 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap starting.
03:07:36.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.138068 13260 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.131:41163
03:07:36.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.138193 13414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.131:41163 every 8 connection(s)
03:07:36.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.139235 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-2/data/info.pb
03:07:36.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.145411 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 13260
03:07:36.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.147858 13285 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Log is configured to *not* fsync() on all Append() calls
03:07:36.151 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:36.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.155196 13417 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:36.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.155467 13417 heartbeater.cc:464] Registering TS with master...
03:07:36.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.156142 13417 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:36.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.157915 13415 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:36.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.158015 13415 heartbeater.cc:464] Registering TS with master...
03:07:36.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.158167 10877 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:36.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.158197 13415 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:36.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.159006 10965 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:36.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.159711 13416 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:36.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.159814 13416 heartbeater.cc:464] Registering TS with master...
03:07:36.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.159999 13416 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:36.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.160051 10965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.131:33837
03:07:36.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.160884 10788 ts_manager.cc:194] Re-registered known tserver with Master: cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163)
03:07:36.162 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:36.163 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.125.130:32851
03:07:36.163 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:36.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.163625 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 12852
03:07:36.174 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:36.174 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] handling channelInactive
03:07:36.175 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state READY due to: connection closed
03:07:36.175 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: connection closed
03:07:36.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.178702 13285 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:07:36.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.179100 13285 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Bootstrap complete.
03:07:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.180539 13285 ts_tablet_manager.cc:1397] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent bootstrapping tablet: real 0.053s	user 0.034s	sys 0.012s
03:07:36.181 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.182 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.182 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test)) after lookup
03:07:36.183 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.183252 13285 raft_consensus.cc:348] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:36.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.183770 13285 raft_consensus.cc:729] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd23193b2ab54c399c0ceb3bf17f7edd, State: Initialized, Role: FOLLOWER
03:07:36.184 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.184 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.184319 13285 consensus_queue.cc:260] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:36.184 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.185258 13415 heartbeater.cc:503] Master 127.10.125.188:32999 was elected leader, sending a full tablet report...
03:07:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:36.185369 13285 ts_tablet_manager.cc:1428] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
03:07:36.201 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.201 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.202 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191))) after lookup
03:07:36.203 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.203 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.204 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.204 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.221 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.222 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.222 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [30ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [30ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837))) after lookup
03:07:36.222 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.223 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.223 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.223 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.321 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.322 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.322 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=147), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [30ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [30ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [49ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [49ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [147ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573))) after lookup
03:07:36.324 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.325 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.325 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.325 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.361 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.362 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.362 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=187), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [30ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [30ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [49ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [49ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [147ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [149ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [151ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [151ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [187ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874))) after lookup
03:07:36.363 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.363 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.364 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.364 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.941 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:36.941 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115),30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)[L],cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:36.942 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=766), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [30ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [30ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [49ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [49ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [147ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [149ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [151ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [151ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [187ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [188ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [190ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [190ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [764ms] refreshing cache from master, [764ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [766ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874) -> (continuation of Deferred@297506394 after retry RPC@763088963), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874) -> (continuation of Deferred@297506394 after retry RPC@763088963))) after lookup
03:07:36.942 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] connecting to peer
03:07:36.943 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.944 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] cleaning up while in state CONNECTING due to: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:36.944 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851) for tablet 7c333092d11b4b969c4a8b742048aeaf: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851
03:07:37.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.321748 13429 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:07:37.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.322063 13429 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:37.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.323364 13429 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851), 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129:46115)
03:07:37.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:37.324456 13282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:37.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:37.325518 13282 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:37.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.328229 13154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" is_pre_election: true
03:07:37.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.328847 13281 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, 30d0e1e440184da1bb263d794b256d01
03:07:37.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.329200 13429 raft_consensus.cc:2726] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
03:07:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.437465 13434 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:37.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.437611 13434 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:37.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.438789 13434 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:37.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:37.439617 13080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:37.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:37.440505 13080 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.444145 13358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" is_pre_election: true
03:07:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.444368 13358 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 9.
03:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.445148 13078 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.445466 13434 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Leader pre-election won for term 11
03:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.445556 13434 raft_consensus.cc:482] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30d0e1e440184da1bb263d794b256d01)
03:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.445600 13434 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 10 FOLLOWER]: Advancing to term 11
03:07:37.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.446583 13434 raft_consensus.cc:504] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:37.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.446997 13434 leader_election.cc:288] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 election: Requested vote from peers cd23193b2ab54c399c0ceb3bf17f7edd (127.10.125.131:41163), 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:37.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.447499 13358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c333092d11b4b969c4a8b742048aeaf" candidate_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd"
03:07:37.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.447700 13358 raft_consensus.cc:3037] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 9 FOLLOWER]: Advancing to term 11
03:07:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:37.448135 13080 leader_election.cc:334] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111)
03:07:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.449846 13358 raft_consensus.cc:2454] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d45cdc346ff4fe4ad8e957a8ad1de47 in term 11.
03:07:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.450201 13078 leader_election.cc:302] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d45cdc346ff4fe4ad8e957a8ad1de47, cd23193b2ab54c399c0ceb3bf17f7edd; no voters: 30d0e1e440184da1bb263d794b256d01
03:07:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.450409 13434 raft_consensus.cc:2781] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 11 FOLLOWER]: Leader election won for term 11
03:07:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.450629 13434 raft_consensus.cc:686] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [term 11 LEADER]: Becoming Leader. State: Replica: 1d45cdc346ff4fe4ad8e957a8ad1de47, State: Running, Role: LEADER
03:07:37.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.450937 13434 consensus_queue.cc:237] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } }
03:07:37.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:37.453502 10965 catalog_manager.cc:5495] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 reported cstate change: term changed from 10 to 11, leader changed from 30d0e1e440184da1bb263d794b256d01 (127.10.125.130) to 1d45cdc346ff4fe4ad8e957a8ad1de47 (127.10.125.129). New cstate: current_term: 11 leader_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d0e1e440184da1bb263d794b256d01" member_type: VOTER last_known_addr { host: "127.10.125.130" port: 32851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d45cdc346ff4fe4ad8e957a8ad1de47" member_type: VOTER last_known_addr { host: "127.10.125.129" port: 46115 } health_report { overall_health: HEALTHY } } }
03:07:38.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.038290 13358 raft_consensus.cc:1261] T 7c333092d11b4b969c4a8b742048aeaf P cd23193b2ab54c399c0ceb3bf17f7edd [term 11 FOLLOWER]: Refusing update from remote peer 1d45cdc346ff4fe4ad8e957a8ad1de47: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
03:07:38.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.039127 13454 consensus_queue.cc:1035] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd23193b2ab54c399c0ceb3bf17f7edd" member_type: VOTER last_known_addr { host: "127.10.125.131" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
03:07:38.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.049734 13080 consensus_peers.cc:487] T 7c333092d11b4b969c4a8b742048aeaf P 1d45cdc346ff4fe4ad8e957a8ad1de47 -> Peer 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851): Couldn't send request to peer 30d0e1e440184da1bb263d794b256d01. Status: Network error: Client connection negotiation failed: client connection to 127.10.125.130:32851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:07:38.421 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 7c333092d11b4b969c4a8b742048aeaf for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297' with partition [<start>, <end>)
03:07:38.422 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=7c333092d11b4b969c4a8b742048aeaf@[1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)[L],30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851),cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)]}]
03:07:38.423 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=7c333092d11b4b969c4a8b742048aeaf, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2247), Traces: [0ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [1ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [7ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [8ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [10ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [10ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [26ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [28ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [30ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [30ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [48ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [49ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [49ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [147ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [149ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [151ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [151ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [187ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [188ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [190ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [190ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [764ms] refreshing cache from master, [764ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [766ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, [768ms] sending RPC to server 30d0e1e440184da1bb263d794b256d01, [770ms] received response from server 30d0e1e440184da1bb263d794b256d01: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [770ms] delaying RPC due to: Network error: Failed to connect to peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851): Connection refused: /127.10.125.130:32851, [2245ms] refreshing cache from master, [2245ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.125.188:32999, [2247ms] Sub RPC GetTableLocations: received response from server master-127.10.125.188:32999: OK, deferred=Deferred@1911675427(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/966761625@62240a62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874) -> (continuation of Deferred@297506394 after retry RPC@763088963) -> (continuation of Deferred@2056073296 after retry RPC@151461514), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@77ed4a2f -> wakeup thread Time-limited test -> (continuation of Deferred@514234045 after retry RPC@133366191) -> (continuation of Deferred@1016096313 after retry RPC@603059837) -> (continuation of Deferred@785885052 after retry RPC@2057638573) -> (continuation of Deferred@1335609749 after retry RPC@947620874) -> (continuation of Deferred@297506394 after retry RPC@763088963) -> (continuation of Deferred@2056073296 after retry RPC@151461514))) after lookup
03:07:38.423 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] connecting to peer
03:07:38.424 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] Successfully connected to peer
03:07:38.440 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x087f26f4, L:/127.0.0.1:58992 - R:/127.10.125.129:46115] using TOKEN/null
03:07:38.457 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.125.130:32851
03:07:38.457 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.458145 10742 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:38.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:38.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.577963 13473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.578270 13473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.578323 13473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:07:38.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.582432 13473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:07:38.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.582571 13473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.125.130
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.587356 13473 tablet_server_runner.cc:78] Tablet server non-default flags:
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.125.148:46139
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:07:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.125.130:32851
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.125.130
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42263
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.125.190:42983,127.10.125.189:33083,127.10.125.188:32999
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.13473
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.125.130
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/logs
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
03:07:38.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 02:22:35 UTC on ccycloud.quasar-gitnpj.root.comops.site
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581827
03:07:38.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.588860 13473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:07:38.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.590421 13473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:07:38.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.600482 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:38.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.600448 13478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:38.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 03:07:38.600539 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:07:38.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.600764 13473 server_base.cc:1034] running on GCE node
03:07:38.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.601372 13473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:07:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.602275 13473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:07:38.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.605618 13473 hybrid_clock.cc:648] HybridClock initialized: now 1786504058605580 us; error 46 us; skew 500 ppm
03:07:38.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.608157 13473 webserver.cc:460] Webserver started at http://127.10.125.130:42263/ using document root <none> and password file <none>
03:07:38.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.608773 13473 fs_manager.cc:359] Metadata directory not provided
03:07:38.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.608852 13473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:07:38.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.613131 13473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
03:07:38.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.615640 13488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.617046 13473 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.617149 13473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data,/tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "30d0e1e440184da1bb263d794b256d01"
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 03:07:05 on dist-test-slave-5k54"
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.617703 13473 fs_report.cc:383] FS layout report
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/wal
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/data
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:07:38.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:07:38.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:07:38.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.640313 13473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
03:07:38.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.641058 13473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:07:38.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.641219 13473 kserver.cc:163] Server-wide thread pool size limit: 3276
03:07:38.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.641768 13473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:07:38.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.643011 13496 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
03:07:38.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.647056 13473 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
03:07:38.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.647123 13473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
03:07:38.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.647176 13473 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
03:07:38.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.648923 13473 ts_tablet_manager.cc:610] Registered 1 tablets
03:07:38.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.649000 13473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
03:07:38.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.649183 13496 tablet_bootstrap.cc:492] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Bootstrap starting.
03:07:38.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.659500 13473 rpc_server.cc:307] RPC server started. Bound to: 127.10.125.130:32851
03:07:38.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.659596 13616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.125.130:32851 every 8 connection(s)
03:07:38.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.660746 13473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF7HWRV/test-tmp/mini-kudu-cluster2773473937275273923/ts-1/data/info.pb
03:07:38.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.666308 10742 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu as pid 13473
03:07:38.666 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
03:07:38.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.671312 13496 log.cc:824] T 7c333092d11b4b969c4a8b742048aeaf P 30d0e1e440184da1bb263d794b256d01: Log is configured to *not* fsync() on all Append() calls
03:07:38.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.672073 13618 heartbeater.cc:347] Connected to a master server at 127.10.125.188:32999
03:07:38.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.672739 13618 heartbeater.cc:464] Registering TS with master...
03:07:38.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.673457 13618 heartbeater.cc:511] Master 127.10.125.188:32999 requested a full tablet report, sending...
03:07:38.676 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:38.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.677367 10966 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:38.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.678382 10966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.125.130:47291
03:07:38.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.681155 13619 heartbeater.cc:347] Connected to a master server at 127.10.125.190:42983
03:07:38.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.681262 13619 heartbeater.cc:464] Registering TS with master...
03:07:38.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.681492 13619 heartbeater.cc:511] Master 127.10.125.190:42983 requested a full tablet report, sending...
03:07:38.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.682472 10788 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:38.684 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786504026297, tablet=7c333092d11b4b969c4a8b742048aeaf, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
03:07:38.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.686360 13620 heartbeater.cc:347] Connected to a master server at 127.10.125.189:33083
03:07:38.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.686452 13620 heartbeater.cc:464] Registering TS with master...
03:07:38.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.686591 13620 heartbeater.cc:511] Master 127.10.125.189:33083 requested a full tablet report, sending...
03:07:38.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.687608 10877 ts_manager.cc:194] Re-registered known tserver with Master: 30d0e1e440184da1bb263d794b256d01 (127.10.125.130:32851)
03:07:38.691 [DEBUG - main] (Connection.java:575) [peer master-127.10.125.189:33083(127.10.125.189:33083)] disconnecting while in state READY
03:07:38.694 [DEBUG - main] (Connection.java:575) [peer cd23193b2ab54c399c0ceb3bf17f7edd(127.10.125.131:41163)] disconnecting while in state TERMINATED
03:07:38.694 [DEBUG - main] (Connection.java:575) [peer master-127.10.125.188:32999(127.10.125.188:32999)] disconnecting while in state READY
03:07:38.694 [DEBUG - main] (Connection.java:575) [peer master-127.10.125.190:42983(127.10.125.190:42983)] disconnecting while in state READY
03:07:38.695 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.10.125.190:42983(127.10.125.190:42983)] handling channelInactive
03:07:38.696 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.10.125.190:42983(127.10.125.190:42983)] cleaning up while in state READY due to: connection closed
03:07:38.696 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.10.125.189:33083(127.10.125.189:33083)] handling channelInactive
03:07:38.697 [DEBUG - main] (Connection.java:575) [peer 30d0e1e440184da1bb263d794b256d01(127.10.125.130:32851)] disconnecting while in state TERMINATED
03:07:38.697 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.10.125.189:33083(127.10.125.189:33083)] cleaning up while in state READY due to: connection closed
03:07:38.697 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.10.125.188:32999(127.10.125.188:32999)] handling channelInactive
03:07:38.697 [DEBUG - main] (Connection.java:575) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] disconnecting while in state READY
03:07:38.697 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.10.125.188:32999(127.10.125.188:32999)] cleaning up while in state READY due to: connection closed
03:07:38.698 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] handling channelInactive
03:07:38.698 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 1d45cdc346ff4fe4ad8e957a8ad1de47(127.10.125.129:46115)] cleaning up while in state READY due to: connection closed
03:07:38.699 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
03:07:38.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.700253 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 13058
03:07:38.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.714445 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 13473
03:07:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.726210 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 13260
03:07:38.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.735066 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 10754
03:07:38.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.742650 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 10841
03:07:38.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 03:07:38.750625 10742 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF7HWRV/build/debug/bin/kudu with pid 10924
03:07:38.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T03:07:38Z chronyd exiting

Time: 36.466

OK (1 test)

